What level of data do you need?
When collecting data, you want to do so as efficiently as possible while still getting the answers you need. How do you manage your research to be cost-effective and complete, without cutting the wrong corners?
It’s critical at the onset of a project to carefully consider what level of data you need. One way to approach this is to think through the unit of analysis you will be using. The unit of analysis is the most important – or smallest – piece of your research and evaluation puzzle. It is the foundation upon which the rest of your project will be based.
For instance, your unit of analysis might be individuals. Or, your unit of analysis might be groups, neighborhoods, cities or states. It’s important that you are clear on this point before you begin, as it is vital that you collect all your data at that level.
Collecting data at the level that is useful to you
It is generally better to collect data at a more granular level—e.g., at the individual level—and then aggregate it to the district or program you’re interested in, but this will depend on your organization and capabilities. To determine what data will be most useful to you, you must think carefully about the ways the information will ultimately be aggregated, reported, and acted upon when setting up your data storage systems.
For example: in an evaluation of the effect of a certain program on villages, the unit of analysis might be villages. However, if you want to know how the program impacted women within the villages, then your smallest unit of analysis must be the individual. If you collect data only at the village level, you will not be able to answer questions (such as those regarding gender) with smaller units of analysis. Village-level data can’t be used to answer questions about individuals, but data collected at the individual level can be applied to both the individual and the village level. Data can be analyzed at the level of the base unit– and at all larger levels. But data cannot be analyzed at any level smaller than the unit of analysis.
Beware of these common fallacies
Don’t let yourself fall into one of these all-too-common misunderstandings of your data.
The ecological fallacy occurs when you make conclusions about individuals based on an analysis of group data. To illustrate this, let’s assume you measured math scores in a particular classroom and found they had the highest average score in the district. Later (probably at the mall) you run into one of the kids from that class and think, “Hey there goes a math whiz!”
Aha! Ecological fallacy! Just because she comes from a class with the highest average doesn’t mean that she herself is a high-scorer. She could have the lowest math score in a class that otherwise consists of math geniuses!
An exception fallacy is sort of the reverse of the ecological fallacy. It occurs when you reach a group conclusion on the basis of exceptional or individual cases. This is the kind of fallacious reasoning that is at the core of a lot of sexism and racism. The stereotypical example of this is the guy who sees a woman make a driving error and concludes that “women are terrible drivers.” Wrong! Exception fallacy!
Pay attention to your unit of analysis
Don’t start your project without a clear vision of the answers you need, and at what level. Remember that you cannot make conclusions about any level smaller than your unit of analysis, so you must collect your data at the lowest level where answers are required.
And when it comes time for analysis of your data, remember to avoid the fallacies that lead many people to erroneous conclusions. This is the foundation of your research and is where the quality of your data collection and analysis will ultimately shine!