On Research Podcast – Survey Research as a Method: Part 2

Darren Gaddis: From CITI Program, I’m Darren Gaddis and this is On Research. Today I spoke with Matt Jans, Lead Statistician for the National Health and Nutrition Examination Survey at the National Center for Health Statistics.

As a reminder, this podcast is for educational and entertainment purposes only. It is not intended to provide legal advice or guidance. You should consult with your organization’s attorneys if you have questions or concerns about relevant laws and regulations discussed in this podcast. Additionally, the views expressed in this podcast are solely those of the guests. I do not represent the views of their employer.

Hi Matt, thank you for joining me again today.

Matt Jans: Thanks for having me.

Darren Gaddis: Last month we discussed consent, what is survey research and what differentiates maybe a good question from a bad question in survey research. Today, let’s pick things up by talking about rigor and how do you establish rigor within a research study when utilizing survey research as a method?

Matt Jans: I’m really glad you’re asking about this because it’s something that our professional field is developed around. As a methodologist, all we do is study methods and we want to figure out the best ways to measure. There’s no one right way, but there are ways that produce more or less bias. So it’s similar to other scientific fields or scientific tools that you might use. It’s always important to use best practices that are out there in the literature.

So I would encourage people listening to this to look up the American Association for Public Opinion Research, or the American Statistical Association, or the Insights Association. Those are three of the big survey and statistical market research organizations that put a lot of time and effort into establishing best practices. Their members have to sign codes of ethics, things like that. Just, if you’ve never done a survey before, if you’ve done them in a context without a framework like that, just orienting yourself to the best practices of those organizations will be a lot. We’ve tried to incorporate those into the course.

Related to that, there are journals out there on survey methods through APOR or ASA, other places, and looking through the most recent research on the topic you’re trying to study to see which methods are shown to produce the most reliable results, is a good idea. That’s the academic framework of it.

A lot of us who work in survey research don’t have the time to spill that whole case. APO and ASA and Insights Association are really good about summarizing the nuggets from the academic field and turning those into best practices. APOR in particular, that’s the organization I know the best, has a lot of published reports and guidelines on major topics in the field.

So for example, over the past couple of decades, the random digit dial telephone portion of survey research has changed quite a bit from only using landline cell phones to also sampling, sorry, landline telephones, only using landline telephones, to sampling cell phones too. So APOR’s put out reports on that and what best practices are. Now that both those kinds of telephone surveys aren’t working so well, in terms of response rates, they’ve put out a report on switching from phone survey modes to self-administered modes or address based sample modes, things like that.

Always good to check those things along with the academic journals that goes with those.

A lot of the time when we’re putting together questionnaires and we don’t have time to build from scratch, we’re pulling from other questions or sampling techniques that have been used in previous surveys or gold standard surveys. So it’s very common for people, at least in my world of government surveys, to look at what the Census Bureau does, partly because whatever the Census Bureau has done to get their statistics on race, education, income, health insurance, whatever, if you’re on the statistical side, once you have your data back from a survey, you’re probably going to do some statistical adjustment, and it’s probably going to be based on census data. So the closer you word your questions to census questions on the dimensions that you’re going to adjust, the better off you’re going to be later.

So a perfect example is race and ethnicity questions. There’s a certain way that race and ethnicity is asked in government surveys, and it’s actually changing right now, so it’s a very interesting area to follow. But if you look at it from a measurement perspective outside of the government survey and statistical adjustment framework, you could look at these questions and say they’re pretty simple. They don’t really capture the breadth of racial identity, ethnic identity and all the things around that. But if you think about it from an adjusted questions and the categories that the government uses, and they’re established and literally codified in a way that every agency has to measure them basically the same way. If you’re going to take estimates from the American Community Survey that the Census Bureau conducts and adjust your survey to that survey, you’re going to have a much easier time if you word your race and ethnicity questions the same way. You could always ask more questions and expand, but as long as you can get your racial and ethnic categories to fit back into that government framework, you’re going to have an easier time.

So let’s see, we’re talking about best practices. It’s always a good idea to pilot your instrument with your population or some version of your population. Sometimes that’s a small subset that you hand pick for a more qualitative pilot, but I always also recommend that people do what we call a dress rehearsal. So where all the pieces are running together, including the sample design, and it’s literally a soup to nuts of what you will do when you have your 1,000 people, you might do it on 100 people as a pilot, dress rehearsal pilot.

I also, it’s real easy. I know we’re going to talk in a minute about some mistake that I see people make, but this one’s not so much a mistake but it’s just easy for smart people and researchers who have done other kinds of research to think that it’s easy to create a questionnaire and a sample and do a survey themselves. But I always recommend that people hire some kind of expert in survey methodology, whether it’s for the sample design or the questionnaire or both, because we have that unique training, we have that constant everyday experience and exposure to different surveys and different contexts. We’re keeping up with the literature and that’s hard to replicate even if you’re a researcher that does a lot of research, but it’s on a different topic. While it’s not rocket science, but it also is real easy to make accidental mistakes that can have big implications. For example, the double barreled questions that I mentioned earlier. They’re real easy to write if you’re not thinking about that while you’re writing survey questions. We’ll probably come back to some of these as we talk more.

Darren Gaddis: On the flip side of that, as you just mentioned, what are some common mistakes researchers should avoid when utilizing surveys as a method?

Matt Jans: Yeah, it’s funny, when I was sketching out my notes for our chat, first thing I wrote was not hiring experts. There’s a lot of different ways that you can hire a survey expert. You can hire them one-on-one consulting, I’ve done that kind of work. You can hire a survey company to run your survey, soup to nuts. Survey companies vary in size and expertise, but there’s lots of different ways to incorporate the scientific and experience knowledge that’s out there. The other thing I wrote, which we just talked about, was not pretesting. You have to have some kind of pretest to make sure your survey is running.

So one common, and again, this might not be a mistake, but it’s a misconception when we do a survey, we usually calculate a response rate. There’s some real technical definitions of what’s considered a response to a survey and what’s considered a response rate, both what goes in the numerator of that rate and the denominator of that rate. It’s not just as simple as the number of people that answered a question in your survey out of the number of people that you thought we would answer a question or survey or the number of people you tried to contact. APOR that I’ve mentioned before, has a set of standard definitions that we all use that includes definitions of what counts as a complete in different modes, survey modes, like a web survey, phone survey, in-person survey, and how to calculate response rates. So I always recommend people look at that. But I’ve seen response rates calculated all kinds of different ways, and they’re not necessarily mistakes, but there is a best practice out there for people to use.

Then the other aspect of response rates, and it’s unfortunate to admit, but my field has done such a good job of formalizing response rates and getting journals to follow these codified standards that people really latch onto them as a quality metric. But it’s easy to confuse a response rate as an indicator, or it’s easy to confuse a response rate with an indicator of non-response bias. Those are two related but very different things. There’s a lot of research out there and even just in statistical theory and principles in general, that response rates do not necessarily mean that a survey has a non-response bias in it, so a low response rate, for example. Surveys of all kinds are suffering from low response rates. You’ll see in our course, and if you just Google this one, you can find all kinds of presentations usually from APOR and ASA that show response rates from almost every survey that reports to them declining over the last couple of decades. So we’re all experiencing lower response rates.

So now the question is, which of our estimates from those surveys actually have more response, more non-response bias in them because of the low response rates? So I always encourage people to untangle those two things and not just worry about their low response rate, meaning that their whole survey has bias because it usually doesn’t.

Then when we think about survey questions, guess we’ve talked a little bit about these. We’ve talked about the encoding topic. We talked about not being overly burdensome with questions or interviews. So almost everybody I’ve consulted with and even surveys where I’ve written the question myself, we want to measure more than we have time for and we end up having to cut questions when it gets close to fielding time or pre-testing time. So there’s a couple aspects of survey burden, but that would be the burden of how much time are you actually spending with this person? How many things are you asking them about?

One thing we didn’t talk about before though, and I see this one a lot, is researchers, usually smart people, very educated, they know their topic that they want to study. Even if they’re not researchers, this might be government folks, policy people, they’re the people that hire you to do a survey. They know a lot about their topic, they know the jargon, they know all the angles and arguments. It’s really hard sometimes for folks to get out of the mindset of knowing all of that, to what the average person that they’re going to sample would know about that topic.

I can think of a transportation and traffic survey I was working on sometime in the last few years where we were trying to write questions about different types of crosswalks that people could cross or different types of intersections. I was learning so much about those things from these great clients and collaborators that I had, but I realized that I had never paid attention to any of this stuff before. I guess now that they’re telling me that there’s these different types of crosswalks, oh okay, I get it, but is the average respondent going to have encoded that because they don’t even know to look for that type of crosswalk? We were trying to figure out how to ask, does your crosswalk in your neighborhood look like this or look like that? We were developing images that we could put in the questionnaire to get the point across to what we were trying to explain. It was a really fun project, but that was a good example of taking this very technical jargon about pedestrian safety and turning it into things that everyday people can relate to.

So a real simple example there was around the term neighborhood. What does it mean to live in a neighborhood? So we say intersections in your neighborhood. If you live in a city style area with blocks and relatively similar size house lots, neighborhoods, probably obvious. I live out in the country, so I look around and I see a couple of houses, but we’re not really on a block and I’m thinking, what’s my neighborhood? I don’t think I really live in a neighborhood per se right now. Some of the things that we take for granted as easy words are a lot more complicated when you start to unpack them.

So one of the pieces that we always look for when looking at questionnaires is making sure that they use plain language. So on one hand, neighborhood might be the right term. You don’t want to get overly burdensome with a definition that says intersections within 100 yards of your house because that’s going to be hard for people to judge. But we want to make sure that how we’re asking questions are intuitive to the general population. We usually talk about writing survey questions at a seventh grade, junior high or middle school level, but just looking for general plain language and avoiding jargon, even if it’s technically correct, even if it’s the right technical term for something in that field. We always have to pretest and see, do average everyday people understand that term the same way or should we use common language?

Then on the sampling and recruitment side, this one, I’ve gotten some pushback on this opinion from people sometimes. I remember doing a lecture at UCLA and the professor’s class I was lecturing for gave me a funny look when I said this, but, I generally advise that people designing a survey just don’t assume that anybody wants to do your survey. That sounds downer but if you go in assuming that your general, particularly if it’s a general population survey, but really for any survey. You’re asking people to give some of their time, nobody has any time to give. You might give them an incentive that would be a good thing, that’s a token of your appreciation, but either way, the survey is your baby. It’s the thing you care about and know about, it’s the thing you’re paying for that you’re going to build a career around or it’s going to help you answer questions in your every day or work life. But the average person who you’re asking to fill it out probably doesn’t care about the topic. Those who do will answer your survey, there’s research on that, but you don’t just want people who are interested in the topic and know about it. You want a little bit of everybody, otherwise you’ll have biased finding. Just a little mental trick that I’ve used for myself is to go into designing a survey assuming that people don’t want to do it.

So then I have to think, it’s always in the back of my mind, how can we make this survey easier or shorter? Or, if I’m giving somebody instructions about how to do it, how can those instructions be clearer? How can we give them an incentive that will be more of a token of appreciation because we’re asking them about sexual behavior or it’s a really long interview? So that’s one aspect of sampling and recruitment that I see sometimes. It’s also just easy to underestimate the time and effort and cost of sampling and recruitment, particularly when you do it in the probability sampling approach like I talked about before. It’s a lot cheaper and easier to just go out and grab 100 people on the street to do a survey of your town, let’s say, than it is to say, okay, I’m going to take all the neighborhoods that live in the town. I’m going to take a map, I’m going to draw a boundary of what that town, the municipality I’m trying to survey. I’m going to figure out a plan that picks at random different neighborhoods to go to.

Then within those neighborhoods, I’m going to pick at random households to go to. I’m going to go away. I’m going to go to the next door that I’ve randomly picked. I’ll record that they said no or told me to go away or they weren’t home, and then I’ll go back again tomorrow and knock on the door. That takes a lot more time. It’s real easy if you haven’t done it for a few years or a few different surveys, it’s hard to see how much time that will take until you actually go through it.

Then lastly, it ties onto that last point. I mentioned earlier that in survey methodology and survey research proper, when we’re talking about our sample, we’re not just talking about the people that we want to and gathered up to answer our questionnaire. We’re talking about defining a population in time and space. I see difficulties that researchers sometimes have if they haven’t defined what that target population is. We usually call it a target population.

So there’s the general population out there, then there’s your target population. Okay, what part of that general population is your target population? Is it people over 18 years old only or is it everybody? Is it people who live in a certain segment of the country or the world? Do they have to have a certain experience? Is it people who are pregnant or might become pregnant? Is it people who have had a certain health condition? If you can define that part really clearly, it makes all the next steps of finding a sampling frame and figuring out how to draw that frame and screening for eligibility a lot easier. If you just jump in and say, okay, we want to survey the whole country. Let’s figure out how to screen people. It’s a lot messier unless you’re… What you should do in that situation, if your target population was the whole country all ages, you say that first. Then if you can say that really clearly, then everything else can fall into place.

Darren Gaddis: With this in mind, how can your CITI Program course, Survey Research: Design, Planning, Implementation and Ethics help someone learn how to do survey research and what could they expect from this course?

Matt Jans: Before I talk about the course, I want to make sure I give a great recognition and thanks to Zoe Paget, who is my collaborator on this course and co-developed with me. We worked ICF together and thought this would be a fun collaboration. She couldn’t make it today.

But our course is titled Survey Research: Planning, Implementation, and Ethics. We try to cover each of those three topics. It’s eight modules long, which may sound like a lot, but we structured it in a way that parallels content on the conceptual or theoretical issues in the framework for understanding survey methodology, followed up by practice and implementation aspects that relate to those conceptual and theoretical issues.

So we organize the whole course around something called the total survey error framework. I’ve hinted at that so far, I haven’t said that phrase yet, but when I talk about there being a questionnaire measurement component to every survey and a sampling and inference component, that’s a total survey error framework. So we introduced that framework and some of the principles and theory and research behind survey methodology, and then go into the practical parts of actually implementing a survey.

Then toward the end of the course, we cover the ethical issues as well, but we get pretty practical at the end where we actually give really concrete ideas about how to develop a survey documentation. We talk about the response rate calculation issue and how to deal with questionnaire specs or specifications. So we take it from the heady level of making sure you have a correct measure to how would you actually implement this with a programmer, whether you’re the programmer or you’ve hired a survey organization or you work in a survey organization and you have to get a questionnaire spec to a programmer. We’re hoping that this covers a wide range of survey experience and student needs.

I guess I can add here too, that survey methodology isn’t a huge field. It’s come a long way in the last few decades and it’s moved a lot more from an art to a science. So we have a lot of research, we’ve used this in the course, that can guide a lot of survey decisions, whether it’s the number of contact attempts to make at a household and the number of times to knock on a door, or call people back, or how to word a question. But we definitely have an answer that we don’t have a study for every single decision that you would need to make or type of question you need to ask. The more survey modes that develop, the less research we have on those new things, on those cutting edge things. So there is still an aspect of survey research that is a craft as much as a science. A lot of the times we’ll have to go back to extrapolating from past research that’s similar but not exactly like the decision we’re trying to make. Work from best practices and come up with good decisions, and we hope that the course will help people with that.

Darren Gaddis: What else should we know about survey research or any closing thoughts you would like to leave us with?

Matt Jans: Sure. One of the things I’ve really liked about working in survey research, I always advocate people, go into this. I think it’s one of these little niche fields that nobody knows exists. Nobody grows up wanting to be a survey methodologist, but it can be a lot of fun and I really enjoyed working on surveys of all kinds of topics, cancer patients. One of the first surveys I worked on was college alcohol consumption. Another was supply chain management. I’ve worked on surveys of child trafficking in Africa, all kinds of topics, and I learn so much each time. Even pedestrian safety I was talking about earlier. So I like being able to learn a lot about different topics and then take what I’ve learned and studied about methodology and make sure that the studies I’m working on are as best as they can be, and so that the research that comes out is actionable, helps policy, helps move the science forward.

Talked a little bit about that issue of it’s still being partly an art and a craft. If you’re going to partner with others, try to find people who have the scientific knowledge about survey methods, but also have a fair amount of experience. So they’ve seen different kinds of surveys and can help you with practical problems as well as the academic ones. I think that’s probably what else to know about survey research. Let’s see.

Oh, so what you’ll see out there, we talk about this in the course, is there is… we try to set the course up as not covering everything that you would need to know to necessarily do a survey yourself, but giving you the broadest perspective of all the issues that you need to consider, and then avenues to go out from our course for more training.

So I got my PhD in survey methodology. You can still get a PhD or a master’s degree in survey methodology or closely related fields like statistics and data science. Data science is definitely the newest field that’s come out in the last 10 years that’s related to survey methodology and it’s taken over the world so much and in a good way actually, I think. But the programs that Zoe and I did our degrees in have now restructured themselves to be survey and data science instead of just survey methodology. They’ve always been a mix of statistical and social science. When I did my PhD, I took courses in social psychology and I also took courses in biostatistics and sampling and things like that. They’ve started to incorporate more, not started to, they’ve renamed and restructured the curriculum, the whole thing, to incorporate more of the data science aspect of things because that’s where training and job needs are going. It’s definitely an evolving field and it’s a lot of fun to work in.

Darren Gaddis: Matt, thank you for joining me today.

Matt Jans: Thanks for having me, it’s been great.

Darren Gaddis: Thank you for listening to today’s episode and be sure to follow, like, and subscribe to On Research with CITI program to stay in the know. If you enjoyed this podcast, you might also be interested in other podcasts from CITI program, including on campus and on tech ethics. Please visit CITI program’s website to learn more about all of our offerings at citprogram.org. I also invite you to review our content offerings regularly as we are continually adding new courses, subscriptions, and webinars that may be of interest to you, like CITI program’s Survey Research: Design, Planning, Implementation, and Ethics course. All of our content is available to you anytime through organizational and individual subscriptions.