We interviewed Stephen Bell, Head of Evaluation Research at Westat, on where he and the research community at large stand on the matter of qualitative vs. quantitative research. Bell gives us much to consider in anticipation of his talk at our June 5th #APPAMWestat forum:
You’re an economist by training. Do you value quantitative over qualitative because of your background?
SB: Quantitative analysis puts a scale on the phenomenon being investigated, so that is appealing. It goes beyond “of what nature” to “how much?” But only where the “of what nature” picture can be captured by variable definitions applied to quantitative information—which is not always the case. Measures of connection among factors also have tighter meaning when quantitative indicators are used, which appeals to economists. Yet, the ability to posit what might be connected—so it can then be investigated—flows best from qualitative information. So with time I have come to see that we need both types of data and analysis.
Where does the skepticism for adoption of mixed methods come from?
SB: Most likely, researchers who prefer to “lead with the qualitative story” feel that moving on to mixed methods adds complications and mixed messages to the findings, needlessly. Or at least they doubt researchers know how to put together the combined qualitative/quantitative story well enough to make it illuminating. Similarly, researchers who prefer to 'lead with the quantitative story' see complications of questionable interpretive value when mixed methods are brought in from the other side. These aren’t good reasons to avoid using mixed methods in policy evaluation. They simply show that it will take hard work and thoughtful integration.
What are some of the deficiencies in quasi-experimental impact analysis that mixed methods evaluation can remedy? How is mixed methods evaluation the answer to those deficiencies?
SB: This guy Stephen Bell at Westat says there is a range of information types, qualitative and quantitative, that are currently neglected when devising quasi-experimental impact analyses—data that in every case have something to contribute. Analysis tools for pulling in each information type, moving beyond propensity-matching of participants and comparison group members on background characteristics to better attack selection bias, is the way he foresees mixed methods evaluation progressing.
In three sentences, what do you hope attendees who are not comfortable with both quantitative and qualitative analysis will take away from your presentation?
SB: A sense that it’s time to get over that discomfort. That there’s too much to be gained in applying mixed methods strategies to remain straight-jacketed in either the quantitative methods camp or in the qualitative methods camp. If the profession starts pushing regularly for the benefits of integrating the two camps, there won’t be any “not comfortable with” problem left for us to talk about in five years!
Stephen Bell is scheduled to present Using Mixed Methods Evaluation to Support Better Quasi-Experimental Analysis at 11:15 AM at the #APPAMWestat forum.