Link Search Menu Expand Document
Responsible data management toolbox

6.2 Create and plan

13-Sept-2022 8 mins Share on social networks


TABLE OF CONTENTS

Case study: the hellish questionnaire

The situation

You are part of an NGO’s Monitoring & Evaluation team, and a project manager sends you a 15-page questionnaire without an analysis plan. During the analysis, you realise that not only were a lot of questions actually useless and therefore asked for nothing to populations exceedingly called upon, but that other questions are missing that are necessary in view of your analytical needs, such as disaggregation by village or gender.

What are the potential risks?

  • An erroneous or even irrelevant analysis
  • Over/undervalued or omitted needs
  • Poor relevance and effectiveness of the response provided
  • Unnecessary additional costs, waste of time for teams
  • Beneficiary fatigue, the latter being called upon for long and/or multiple surveys
  • This results in reputational risks for the NGO

What to do?

You will probably not find a silver bullet to remedy this situation and will need to plan a complementary (and probably cumbersome and expensive) data collection for missing data if the latter are a real stumbling block for your operational or accountability and capitalisation needs.

How could the situation have been avoided?

The obvious way to avoid this type of situation is to prepare any collection exercise with an analysis plan that has been discussed between the project manager and the monitoring & evaluation team, specifying the necessary analyses, the disaggregation you need, the collection methods, the biases envisaged etc.

Complementarily, it is essential to support program managers in mastering the importance of the analysis plan to render the collection more effective, as well as the notions of data minimisation, beneficiary fatigue and the limitations of an “I collect today because I may need this data tomorrow” approach, which simply does not respect the principles of data protection, which involve precisely defining the purpose of the collection and prohibiting the reuse of the collected data.

Data collection should be anticipated and prepared, taking into account the context in which it will take place. In particular, if you are in a context with multiple actors carrying out regular surveys, a little coordination can only help respect the people you seek to accompany by soliciting them only for what is necessary).

Case study: the choice of a tool unsuitable for the sensitivity of the data

The situation

A project manager has asked you, in your capacity as Monitoring and Evaluation Manager, to set up a satisfaction survey of populations who have received hygiene kits on a mobile data collection tool. Just before the collection ends, in speaking with colleagues from other organisations having analysed each tool for their protection by design and default features, you realise that the tool was inadequate, with insufficient personal data security and a server located in a country not compatible with the applicable legislation.

What are the potential risks?

  • Personal and sensitive data accessible by third parties
  • Risk of data loss and destruction
  • In the event of audit, risks of fines and thereby reputational risk for the NGO

What to do?

With the collection already half over, it would seem excessive to call everything off and recommend switching to an alternate tool, except if the contextual analysis shows risks of harm to people associated with this collection.

On the other hand, additional data protection measures can be implemented quickly to mitigate the risks involved, such as:

  • the use of business phones and computers only,
  • securing mobile phones (pin codes, password to access the application,
  • software and antivirus updates,
  • regular cleaning of the data sent, of the application (more granular profiles and user rights, restricted access for the project…),
  • the implementation of secure modalities for data sharing (secure Wi-Fi or mobile data, VPN if necessary, use of encryption functions, or even reviewing the form so that the identification data is captured as a unique identifier rather than nominative data).

Keep in mind that you do not bear responsibility for this type of situation and that it is better to think about alternative solutions, which in parallel makes it possible to raise awareness of the issues. In addition, the organisation as a whole must be able to provide you with the means to work properly (guides, policies, tools, road map, etc.) to guide you in the choice of tools.

How could the situation have been avoided?

The ideal situation is when the organisation has made an institutional choice as to which tools to use, and that the latter have been verified on the basis of responsible data management criteria.

Of course, given that the tool is only a variable in the equation, it is important that field teams have standard operating procedures to follow (“Standard Operating Procedures” or “SOP”) on which to rely to avoid errors or omissions.

Secondly, it is apparent that a risk analysis helps guide the level of data protection to be implemented to ensure a collection that is both secure and fluid in relation to operational issues.

Case study: a poorly planned data collection

The situation

Respondents are subject to data collection, the meaning of which they do not fully understand, between the fact that the questions are not in their main spoken language and that they contain many technical terms or acronyms that enumerators are not always capable of articulating clearly

The answers they give are therefore hazardous, but they do not dare not answer out of respect for the enumerators and for fear of compromising their access to subsequent NGO services.

As program manager, you realise - in analysing the data at the end of the day - that there are discrepancies, and you get closer to the M&E team to figure out the problem. The latter reminds you that you gave them 24 hours to design the collection tools and translate them into the desired languages, which was clearly not enough to obtain a good quality collection.

What are the potential risks?

  • Wrong analysis; even off-topic
  • Over/undervalued or omitted needs
  • Poor relevance and effectiveness of the response provided
  • This results in reputational risks for the NGO
  • Poor use of resources, waste of time, frustration of teams and enumerators

What to do?

If the data collection is of a certain magnitude, very important for operational needs and the discrepancies identified are significant, the best option to pause the data collection:

  • Until the enumerators have been properly trained on the content and objectives of the survey,
  • and the tools revised to be more explicit by taking into account feedback from the field,
  • and this, with a quality translation therefore tested beforehand with the local team also in the languages spoken by the people surveyed.

How could the situation have been avoided?

It is common that problems emerge during collection due to lack of preparation. It is therefore essential to set up procedures between thematic and M&E teams to ensure that everyone can set up the collection tools in good conditions, following established steps.

It is often necessary to raise awareness among program managers of the risks of allowing too little time for the teams to design, translate, and test, which is often time-consuming depending on the complexity of the form envisaged.

It is essential that the collection teams are trained on all the dimensions of the survey (objectives, vocabulary, conduct, consent, respect of the respondents…) with, if possible, simulations / coaching in the first days to identify potential problems.

Key resources