Thinking Like a Participant

I find that the most creatively intensive part of most social scientific endeavors are in the very early stages of a project. Some projects have an obvious starting place, like when a massive research gap in previous research is staring you in the face or when you are building on a previous project in a natural extension. I don’t think this is the case for most studies, however. Many studies can start from a central idea, but require you to narrow down from a myriad of questions or competing ideas, and develop an instrument that is actually cohesive and understandable from the perspective of your participants. At least in my experience, the basic germ of an idea comes early and can be articulated in a single sentence. What is more complicated is turning that idea into something workable. In an interview-based study that led as a graduate student, “A View from the Trenches,” the germ of this idea was: how do journalists who cover science perceive and cope with changes to the wider science media ecology?

When I put it like that, it sounds nice and clean and even likely understandable to people that are outside of the field. But as a research question it is unfocused and unworkable. Pose that question to participants of interest and expect to get about as many different responses. Reasonable answers to this question could be: “the loss of respect for the profession is unnerving,” “I turn off my social media notifications at home,” or even “what do you mean by ‘media ecology?’” Clearly, such varied answers are problematic to make sense of in qualitative data analysis. While some problems are fairly obvious to the astute researcher, after all the question is clearly double-barreled and quite possibly leading the participant to see problems in the media ecology, it illustrates the difference between the idea that inspires the research and the actual substance of the project from a participant’s perspective. The bridge between these two steps is the phase of the project that usually requires the most creative input from me.

Returning to the example of my project with my PhD advisor, Anthony Dudo, the germ of the idea essentially came right away. It was clear from the existing literature that there were indeed changes in the science media ecology and that they drove adaptation strategies on the part of journalists who write about science. In fact, we use the phrase “journalists who write/publish/cover (about) science” to distinguish our target population from science specialists who have all but ceased to exist following downsizing on the part of traditional news outlets. At this stage, we had some particular changes to the profession in mind and we are on the way to developing a specific study, but it would still be too broad at this point. This was not necessarily theory-driven research in the large scope. That is, we did not have previous studies to model after to the degree that we would be able to have pre-set questions or relationships to test. Our goal with this was exploratory in nature. Thus, we had to take the mass number of previous publications around the structural changes in science journalism and distill them to a research instrument that could be adequately used in the course of hour-long interviews. This narrowing is inherently a creative process for a social scientist, because they need to make design choices under which they have creative freedom.

I think a good way to think about these choices is to categorize them into whether they flow from previous literature or transcend it. As a general rule, most study designs in theory-driven research flow from previous studies. In such projects, you may be applying what is known in a theory to a new population or context, or may even be attempting a direct replication of a previous study. That said, theory-driven research often has transcendent components. Consider a theoretical extension in which you add a new variable to an existing theoretical model. Before you thought this up, it did not exist. But why pick that variable? Why not include every variable that you could possible imagine being relevant to the theoretical system you are interested in? You are constrained if nothing else by the study design itself. Put simply, more variables means more questions means greater length of the instrument. There aren’t all that many study participants out there that will tolerate an hour long online questionnaire, or a three hour long interview. So more likely than not, as a social scientist, you will need to make some choices that narrow your broad idea down to a specific study design. You will need to use your judgment to make those choices in the most meaningful ways. Even after this, you still do not yet have a design that is ready to be fielded.

What comes next in designing a project is to actually envision the study. You know what the study is trying to answer, but what are you actually doing? I learned this in a simple way from an interaction with a mentor during my Master’s program, Mike Xenos. I had in mind an experiment that I intended to run for my thesis, but my thoughts were entirely on what it was supposed to tell me (e.g. “how effective are podcasts for communicating science?”). He asked me what I was planning to do in my study at a department social and I began to rearticulate the literature review I had conducted on the topic, talking about the audio nature of the medium and the relative literature gap in podcasting compared to text or video social media (I promise this was more novel in 2019 than it would be now). As I talked on, and I watched his reaction, I realize that I never answered his question, because I gave an answer to the why of the study but not the what. His response was pretty simple but actually had a profound effect on my thinking as a social scientist. He asked me to back up and think about it like this: “what is it like to be a participant in Josh Anderson’s experiment?” (quote to the best of my memory)

That simple question is actually a profound pivot in thinking. The biggest thing is that it changes the focus of the actual material of the study away from what you as the researcher want to what your participants give you. Clearly you want these things to align or, stated otherwise, your data should address the motivating questions of the study. But the trick here is that, in social science, your data is exclusively the product of people. If you are collecting the data in the form of survey- or interview-based methods, your data is exclusively from the perspective of your participants. Consider the classic third-person effect: if you ask a participant to tell you about how another person feels about something, they cannot actually tell you what that person feels with any reasonable degree of accuracy. They will be estimating what that other person will feel from their own perspective. The answer can actually be revelatory into how your participant feels that may be less clouded by experimental biases than if they were responding as the self. Think about it this way: any instrument that you present to participants should be oriented toward what they can meaningfully answer.

Returning again to the “Trenches,” like any other semi-structured interview-based study, Anthony and I developed an interview protocol as a list of focused questions that was designed to give some consistency to our participants. While a protocol in this type of study is not typically run through verbatim in any given interview, it is important to word questions in such a way that you get a response in the expected range. In our case, a question like “please tell me about a challenging experience interviewing a scientist” followed by “how about a rewarding experience?” is more workable than “tell me what it is like to interview scientists.” This is because it taps into what we are interested in for the study (challenges and successes of the work as it is currently done) in way that the participants can actually answer.

Understandability is also a large concern. For our study, consider a question like “how do motivated reasoning pathways to misinformation belief influence your work?” While this is a well-understood point in science communication and the subject of studies published in top-tier journals, we wouldn’t necessarily expect journalists to know the finer details of the literature here. to tap into this, we asked multiple successive questions that were similar to: “what do you think of when you here ‘science misinformation’?”, “does your understanding of science misinformation guide your work as a journalist, if so, how?”, and “why do you think people believe science misinformation?”

These questions also illustrate my final point regarding leading questions. As any decent social science training will tell you, the specific wording of questions can affect the answers you get, and a lot of that has to do with presuming a set of answers in the text of the question that may not allow your participant to answer accurately. Consider this instrument: “please tell me how the loss of science specialist positions in recent decades has affected your work as a journalist.” Obviously, this presumes that there has been a loss of specialist positions, which may be forgiven as previous literature has documented this well. But it also presumes that our participant is personally aware of it, which we cannot safely assume. Additionally, it presumes that it has affected their work, which certainly cannot be assured for any given participant. How we might rework this is to ask first about what their experience of the labor market has been: “have you ever sought out a job in a science beat? If so, how easy has it been to find a job like that?” For journalists with a long time in the field, you might then ask how it has changed over time: “compared to previous years, do you think finding jobs on the science beat is easier or harder?” And to top it off you can finish with, “given your thoughts on specialized science beat jobs (asking them to recall their previous answers), does this state of the field affect the work you do? If so, how?”

My central point of this essay is such: the actual instruments of your study as a social scientist need to be oriented toward the perspectives of your participants, and from there answer your core research questions. This is inherently a creative process. As the social scientist designing the study, you have a lot of power in the design that I encourage you not to neglect. You are the person designing the experience of the study participants. You select which specific topics are covered over the course of the study, which requires creative input and good judgment to address and transcend previous research. You also decide how these points are developed into an instrument that your participants experience, and you would do well to think about what your participants can actually answer. When you go through these steps you move your project from the “why” to the “what” stage. A practice that I like to do is to review my study materials thinking from the perspective of a participant. Ask yourself: “would this question make sense if I did not know the literature?”, “is there a way to answer this question correctly that does not give the response that I am looking for?”, and “will participants answer this question in a way that gets at what I’m looking for?” This may not be quite as applicable for research that is not developing new study instruments, as these have likely been used and tested in other contexts, but most research that is not a direct replication has some novel component that warrants thought from the perspective of the participant. For instance applying study questions to a new population requires you to think about how those questions would function in that population. As a final example from “Trenches,” we would expect journalists and scientists to have different bases of knowledge in science journalism, and we would do well to adjust our questions accordingly. Moving beyond question text, the same thinking can be applied to other study elements, such as experimental manipulations.

Speaking to the early career social scientists out there, if you are ever in the position that I was when I spoke with Mike, consider an answer that addresses this: what exactly will your participants see, what you expect to see given your study population, and why chose that design you did. Preferably in that order. If you are able to describe a project as such it will demonstrate you have a good understanding of your research project and you understand what your participants can give you from it.