2.4.3 Building an analysis plan

TABLE OF CONTENTS

• Research plan
• Sampling
  • Non-random sampling methods
    • Self-selection
    • Snowball sampling
    • Quota sampling
  • Random sampling methods
    • Simple random sampling
    • Stratified random sampling
    • The random walk
  • Define the sample size
    • Confidence level
    • Level of accuracy
  • Defining the sample size

---

Research plan

Once you’ve identified the question(s) or problem driving your analysis and the resources that you have available, it is time to pull together an overall analysis plan for how to proceed. Below is a table of the information needed to get you started with creating a research plan. This is just a starting point and you may want to make some modifications to this template depending on your specific context.

This is not always the case, but it is also possible to include in the analysis plan details of how data should be viewed. This can help to project the type of analysis you want, and to anticipate the final output.

Building a research plan will help you to be more aware of the data you already have available and the data you really need. To find out your data needs, you can also resort to part 4 - 1 Making decisions to get the data we need and part 4 - 4 Hands-on Review with External Data sets of the Module 4: Getting the data we need of the IFRC Data Playbook.
Don’t hesitate to also look into the one-page point 1D Developing the analysis plan in the The Kap Survey model - Knowledge Attitude and Practices that gives theoretical and concrete guidelines on analysis plan.

Specific examples on each section are provided in the analysis plan for the case study that can be seen here.

Sampling

To ensure your analysis will be correct, you sample should be representative of what you are trying to study.

• The first thing for that is to know the survey population (could be the total population of a city or a camp for instance), as well as the sampling unit (could be individuals or households for instance)

• Then you need to choose your sampling method: it can be random or non-random. For statistically representative results random sampling is recommended, but not always possible.
  • There are 5 main types of non-random sampling methods that you can use: self-selection, snowball, quota and convenience, purposive sampling (not presented here). But keep in mind, that the representativeness of your sample might not be optimal and might bias your observation.
  • Similarly, there are 3 main type of random sampling that you can use: Simple random sampling, Two stage cluster sampling, Systematic random sampling.

Non-random sampling methods

Self-selection

Principle: Individuals in the target population are invited to complete the survey. They are free to respond or not.

• Advantages: Easy to administer
• Limitations:
  • Difficulty in obtaining a representative sample of the target population.
  • Requires that the entire target population has access to the information and is able to express interest in participating in the survey.
• Potential biases:
  • Over-representation of cooperators.
  • Over-representation of people who have a sensitivity/attraction to the survey topic.

Snowball sampling

Principle:

1. Selection of the first n1 respondents (= germs) from the target population
2. The respondents redirect us to other people = the respondents of the next wave.
3. (… etc …) Ideally : Until the composition of the final sample is independent of the composition of the initial germs.

• Advantages:
  • Allows access to hard-to-reach respondents.
  • Better participation rate (*)
  • More sincere responses (*)
    

(*) Because respondents are identified and approached through a personal network

• Limitations:
  • Difficulty in obtaining a representative sample of the target population.
• Potential biases:
  • Over-representation of people with an extensive social network. Conversely, lack of isolated people.
  • Over-representation of cooperative people.

Quota sampling

Principle: establish quotas of people to be surveyed according to the characteristics and proportions of the population. The aim is to try to give the sample a structure similar to the mother population on certain criteria (normalisation hypothesis).

The quota variables (x1 and x2) must be correlated with the variable of interest. This requires a priori knowledge of the phenomena studied and the main variables that determine behaviour. This method is used when the mother population is well known.

• Advantages: Easy to administer
• Limitation: There is a frequent lack of information about cross-quotas. The interviewer just has to respect the marginal quotas so it is possible to obtain cross-quotas.
• Potential biases: The probability of selection of individuals is unknown. One type of population will remain unsurveyed (selectivity bias).

Random sampling methods

Simple random sampling

Principle: All individuals in the target population have the same probability of being selected into the sample.

• Limitations:
  • Requires an exhaustive list of the entire target population : Very heavy databases to manage.
  • The list must be able to identify and contact individuals without ambiguity.
  • Information must be up to date and of good quality.
  • Geographical dispersion of selected individuals : Additional costs and logistics.
• Potential biases:
  • If the list does not meet the above criteria.
  • If the individuals in the target population have very different characteristics (very heterogeneous target population).

Simple random sampling can be used for in-depth evaluations and especially when :

• There is available data on the population
• All parts of the affected area are accessible
• The situation is reasonably stable
• There is sufficient time to visit all selected households and conduct the required number of interviews.

Stratified random sampling

Principle: Individuals are partitioned into homogeneous subgroups (= strata) (with respect to certain distinct characteristics of the target population). Then simple random extraction for each stratum.

When to use this method? When the individuals in the target population have very different characteristics. In which case, a simple random sample could be catastrophic.

• Advantages:
  • Allows the heterogeneity of the target population to be represented. To do this…the behaviour of individuals in a given stratum should be as similar as possible with respect to the variable of interest.
  • Reflects the true representation of the population - if done well.
• Limitations:
  • Difficulty in choosing the stratification variable.
  • Requires auxiliary information for all units in the target population.
    

Often the stratification variables are qualitative and not quantitative which makes stratification more complex.

The random walk

Principle: The interviewer walks through the survey area following a specific pattern to randomly select the individuals to be interviewed.

• Advantages: No need to have a list of households prior to collection.
• Limitations: Representativeness of the population in case of absence of some households during the survey.

Define the sample size

The sample size varies according to : • The size of the population of interest • The desired level of confidence • The desired level of precision (sampling error)

Confidence level

First of all, you need to define the desired level of confidence. This is the certainty that the results of the sample accurately reflect (are representative of) the entire population under study. The commonly used confidence level is 95%.
A 95% confidence level means that 95 times out of 100, the sample results will accurately reflect the entire population. If evaluators are willing to be 90% sure, they can use a smaller sample. If they want to be 99% sure, they will need a larger sample.

Level of accuracy

It is then important to consider the level of precision of the estimates desired. This is known as the sampling error, or margin of error. This is the average difference between all estimates for all possible samples and the value that would be obtained if the entire population had been surveyed.

Defining the sample size

The sample size is a function of the size of the population of interest, the desired level of confidence and the desired level of precision. The sample size can be determined by

1. Using a formula. The following online calculators allow you to estimate the sample size you need for :
   • Estimating the proportion of a population :
     • indikit
     • select-statistics.co
     • raosoft
   • Estimating the average of a population :
     • select-statistics.co
2. Using a reference table that indicates the sample size needed for a given confidence level:
   

Depending on the sampling method, the sample size, margin of error and confidence level will vary:

• Sample size: number of units surveyed = size of the girl/observed population.
• Confidence level: the certainty that the sample results reflect reality, within the margin of error. A higher confidence level requires a larger sample size.
  • Usually 95%.
    
• Sampling error: margin of error: the average difference between all estimates for all possible samples and what exists in the target/mother population (= actual value - estimated/observed value).
  • Usually 5%.
    

The desired confidence level and precision can be improved by increasing the sample size. The standard to aim for is a 95% confidence level (margin of error +/- 5%). The larger the margin of error, the less accurate the results. The smaller the population, the larger the ratio of sample size to population size.