On Tech Ethics Podcast – Open Science Principles, Practices, and Technologies

To easily navigate through our podcast, simply click on the ☰ icon on the player. This will take you straight to the chapter timestamps, allowing you to jump to specific segments and enjoy the parts you’re most interested in.

1. Introduction to Brian Nosek (00:00:03) The host welcomes Brian Nosek, discussing his background and the focus of the episode on open science.
2. Brian Nosek’s Work (00:01:16) Nosek explains his role at the Center for Open Science and his research focus on implicit bias.
3. Defining Open Science (00:02:43) Nosek outlines the core principles of open science: transparency, sharing, and inclusivity.
4. Stakeholders in Open Science (00:05:42) Discussion on the roles of researchers, funders, publishers, and institutions in promoting open science.
5. Practices for Transparency (00:08:57) Nosek discusses core practices and technologies that enhance the transparency of research.
6. Sharing Research Outputs (00:12:03) Focus on the importance of making data and materials accessible for reuse by the research community.
7. Inclusivity in Research (00:14:28) Nosek explains practices that promote inclusivity, such as participatory research and team science.
8. Challenges in Applying Open Science (00:19:09) Common issues researchers face include lack of knowledge, support, and misaligned incentives.
9. Resources for Open Science (00:22:00) Nosek discusses the Open Science Framework and other resources to help researchers implement open science practices.
10. Registered Reports Publishing Model (00:24:38) Discussion of registered reports, a model that shifts the primary peer review to right after the research design phase.
11. Emerging Trends in Open Science (00:27:10) Overview of the mainstream acceptance of open science and the need for improved quality in shared data practices.

Daniel Smith: Welcome On Tech Ethics with CITI Program. Our guest today is Brian Nosek, who is the co-founder and executive director of the Center For Open Science, and a professor at the University of Virginia, focusing on research credibility, implicit bias, and aligning practices with values. Brian also co-developed the Implicit Association Test, and co-founded Project Implicit and the Society for the Improvement of Psychological Science. Today we are going to discuss the principles, practices, and technologies associated with open science.

Before we get started, I want to quickly note that this podcast is for educational purposes only. It is not designed to provide legal advice or legal guidance. You should consult with your organization’s attorneys if you have questions or concerns about the relevant laws and regulations that may be discussed in this podcast. In addition, the views expressed in this podcast are solely those of our guest.

On that note, welcome to the podcast, Brian.

Brian Nosek: Thanks very much for having me.

Daniel Smith: I’m really looking forward to our conversation today, and learning more about open science. But first, can you tell us more about yourself and your work at the Center for Open Science?

Brian Nosek: Yeah. I am a psychologist by training. I’ve had a lab at the University of Virginia since 2002. The Center for Open Science was spun out of my lab in 2013, as a nonprofit with a mission to increase openness, integrity, and reproducibility of research. We spun it out of the lab because my substantive area had, for many years, been focused on implicit bias, or thoughts and feelings that occur outside of awareness or control, and how those might affect our assessments of ourselves or of other people based on social characteristics or other things. We started to do a lot of projects that were looking at the role of bias in science. And how we might, without realizing it, be influenced to find things that are in our interest, personal interest, career interest, other type of interest, and not necessarily aligned with doing the most credible science possible. Even though that’s our conscious goals, as researchers.

The Center has tried to address the factors, both social and individual, that constrain our ability to do the best work we can. And to try to help change the system, the culture of science, to promote the values of science.

Daniel Smith: Thank you, Brian. Now, I think you alluded to this a bit in your introduction. But just so we’re all on the same page, can you briefly describe what open science is and its core principles?

Brian Nosek: Open science has three general meanings. Openness as transparency, meaning you should be able to see how the research was done, how the researchers drew their conclusions from what they did. Openness as sharing. You should be able to access the data, the materials, the code, the processes that underlie the research, rather than just the researchers report of what they did and what they found. And openness as inclusion, meaning that everyone has a stake in science and scientific progress. The system of science, as a social system, should include everybody in some degree or another. Having community shape what questions are asked, what are the priorities for different areas of the world, of humanity, of types of ways of trying to advance progress.

Can we be more inclusive in who helps shape the direction of scientific questions and investigation? Through being able to contribute to science in a variety of different ways. At the design, at the execution and reporting, in trying to make sense of all of that information. These themes of transparency, and sharing, and inclusion translate into a lot of different practices that comprise how people think about open science. Primarily, they fall into the buckets of who gets to be involved in the research, what is actually being done in the research, and how can it be improved.

The types of practices that are involved in that most commonly are open access. Meaning the more open sharing of the papers, the outcomes of the research so that the funding public that paid for most of that research to be conducted can actually access the research itself, rather than it being protected behind a paywall. Sharing the data, materials, code, all of the content that’s produced during the research process. Having stronger standards for planning, like making data management plans at the outset of research so that you know how the data will be shared once it’s completed. The concept of preregistration, of actually committing to what your design and your plan for analyses are before you observe the results, to try to increase the credibility of the inferences that you draw from those. And a variety of other things that expand innovation, and collaboration, and inclusion of the entire process.

Daniel Smith: We’ll talk a bit more about the core principles and technologies for each of those three areas in a moment. But first, I want to go back to what you said about how everybody has a stake in open science, and hear more of about the different stakeholders involved. Funders, publishers, institutions, societies, researchers, and so on. Can you just talk a bit about the role that each of these groups plays in upholding the principles of open science?

Brian Nosek: Yeah. They’re each critical in different ways.

Starting with researchers, and to some extent societies. A lot of how science is done is created through norms. Researchers observe what others in their area of research, in their discipline, in their department, what they do. They say, “Oh, that’s how I should do research as well.” The opportunity to promote transparency, and sharing, and inclusion comes from trying to shape the norms in different scholarly domains, so that those values are more prominent.

Simultaneously, there are other stakeholders. Funders, publishers, and institutions that can directly shape the reward system that influence how researchers decide to behave in their work. One of the big challenges for the open science movement, and partly why it is so prominent, is that the reward system for science is misaligned in some ways with the values of science, which may be creating some friction in the pace of discovery.

For example, it’s more important for me in advancing my career as a researcher to get things published, and ideally published in a highly prestigious journal, than it is for the actual content to be accurate. Because the publication is the reward, rather than the accuracy of my findings. At least, in the short term. Because of that, there are a lot of dysfunctional behaviors that occur that might lead me, even if I’m not trying to do non-credible research … I didn’t get into research to do things that are not true. I’m trying to discover true things. But nevertheless, because of that reward system that demands publication, and publication in prestigious places, I might unwittingly end up doing things that reduce the credibility of my findings to produce those findings that are more exciting, more publishable, more novel.

The role of funders, and publishers, and institutions is to help shift that reward system so that it compliments the norms that are emerging among researchers, who almost universally support principles of open science. They just see they’re not rewarded. So that, how you get funding for your research is tied to the rigor and transparency of your research practices. How you end up getting your papers published is tied to the rigor and quality of that research. And how you get a job at a university or other research institution, and then get to keep that job, is tied to rigor and quality of the research.

All of these groups simultaneously play substantial roles in shaping the overall reward system of science. None of them individually can change that system on their own.

Daniel Smith: Now that we all have a better understanding of what open science is, its core principles, and the roles of the different stakeholders, let’s get into the core practices and technologies for carrying out open science.

First, what are the core practices and technologies associated with transparency or making research visible?

Brian Nosek: Yeah. The fundamental elements of transparency are how do we go beyond just sharing the finding, the outcome of the research? This is surprisingly a big challenge for the research community because the way in which the business models of science emerged for scholarly communication is that publishers gained access and control over the content of research findings, and provide these to other researchers through subscription models. You have to pay to access the findings.

Now that is a big limitation to what is supposed to be a public good for expanding and sharing, making transparent how it is researchers are doing their research and what they’re learning. That has been changing over time, where governments and individual actors have been shifting the norms to a more open access of at least the findings of research. Terms like green open access, where papers are shared without all of the nice, pretty formatting in the final publication are made available freely is one form of open access, being more transparent with the findings for the whole research community.

And transparency also involves how did you get to those findings? It’s fine that you can show me your findings, but that’s your description of how you did the research and got to the claims that you got to. For me to really evaluate your research, I need to be able to see how you got there. What was the design you used? What kind of data got generated? How did you apply your analysis strategy to those data? If I can’t actually see all of the components of the research that are summarized in that final paper, then it’s very hard for me to evaluate the quality of that research, and to question, and be skeptical of the findings.

Those are really critical aspects of how science makes progress, by rooting out error and pursuing what it is we aspire for science to be, which is self-correcting. The reason that self-correction is so important for science is that science is always pushing the boundaries of knowledge past what we know. If we are doing that, and doing that well, then we’re going to be wrong a lot. Most of the time, perhaps. The ideas that we have, the claims that we have, the initial findings that we have are going to turn out to be totally incorrect, or much more qualified, or explained differently than we first thought. That’s the natural part of science, of pushing the boundaries into the unknown. But for that to work well, you have to be able to self-correct and root out the errors. Without transparency, it’s almost impossible to do that effectively.

Daniel Smith: Certainly. Moving on to sharing, which again, is making research accessible and usable. What are the core practices and technologies associated with this principle?

Brian Nosek: Key part of sharing is let’s not just make the data, and the materials, and the code, and the preregistrations of plans observable to readers and reviewers of the research, but let’s actually make it so they could use it themselves. Reuse it, use it in different ways, extend the work. Really good sharing involves data repositories, and other types of repositories that make available all of these outputs for free, or easy reuse by others in the research community.

Those services play a vital role in making it possible for researchers to take data that I produced in my lab, reanalyze it to see if the findings that I reported are reproducible. So that they can see, “Yeah, I get the same findings that you reported. I don’t find any errors.” Or maybe they take that data, and they analyze it different ways. “Yeah, you said you analyzed it this way, but there are a lot of different decisions you could have made in how you analyzed that data. Let’s see if your findings are robust to those various changes.”

Then finally, they might take that data and try to extend it in a different way, and do something new with it. They might use the materials that I generated to produce my data, and run a totally independent study that builds on or challenges mine, to see if mine replicates, or to see if it holds under different circumstances. Or to test a different reason that the observations might have been observed.

This community of services that are open registries and repositories play a critical function in not just making things transparent, but also making all of this work underlying our findings available to others.

Ed Butch: I hope you are enjoying this episode of On Tech Ethics. If you’re interested in important and diverse topics, the latest trends in the ever-changing landscape of universities, join me, Ed Butch, for CITI Program’s original podcast On Campus. New episodes released monthly. Now, back to your episode.

Daniel Smith: Going back to our final principle, inclusivity, which is involving and crediting more contributors and research. What are the core practices and technologies associated with this principle?

Brian Nosek: There are a few of them. In the most narrow sense, the way that we’ve given credit for research contributions is through authorship on papers. One opportunity that the research is pursuing is expanding the notion of credit to be more precise about the contributions that each researcher made.

In the standard model, the ideal is your first author on a paper or last author, or at least named on the paper. If I’m fifth author on this paper, what does that mean, if it’s 17 different authors. It’s hard to attribute credit. One of the innovations is models like what is called the credit system, which actually has people attribute how they contributed to the research process by enumerating what their contributions actually were. Increasing transparency, and also crediting different types of contributions.

It’s an opening towards more inclusivity to recognize that there isn’t just one way to contribute to science. In fact, there are many ways to contribute to science. This movement to more transparency about contributions is leading to opportunities for greater inclusivity of different types of ways to contribute, and then get credit for one’s contribution to science.

For example, I might be great at generating data, but actually terrible at writing papers. I hate writing papers, so I never write papers. But I generate this data that’s so useful for so many others to do their own reporting and analysis of that data. I should be able to get credit for doing that part of science because it is a very important part. Can we increase inclusivity by expanding the way in which we provide roles and opportunities for career advancement that are more diverse than how science was originally conceptualized?

Then another way that inclusivity gets expanded is by actually engaging a broader community for how it is they can contribute to research. A great example of this is movements towards participatory research. Where, for example, there might be an interest in public health in a particular area of the country. Rather than the public health researchers just going into that region, and doing their research and leaving, they could adopt a participatory model that goes in and talks to people in that community. Identifies the questions that are of interest to the researchers, but then also asks the people in the community, “What are your interests in relation to this question?”

What’s very interesting is that a lot of the experiences with doing that reveal all kinds of questions that aren’t ones that the researchers even considered. Or lots of context that the researchers had no idea that they should consider in relation to public health in that context. Participatory research is really embracing the idea that researchers standing alone in the lab is not sufficient for many of our questions. To actually engage the questions of interest, the questions that will matter, that questions that will have impact. How do we more effectively engage those that can contribute in substantial ways?

A last thing to mention is team science. There’s been a lot of movement towards increasing the opportunities for people to work in aggregate, because many of the questions in science are big questions. They’re not questions that could be investigated productively by any one individual or any one lab. But if many individuals and many labs work together on that problem, they might be able to make a lot more progress by sharing resources, expertise, and dividing up the work more effectively.

There’s great examples of this. For example, in high energy physics. CERN discovering new particles is a massive, massive collaborative team science project conducted in that field. But many other fields have emulated and extended on that model to show that team science, at a large scale, can be very effective. There’s just lots of innovation happening in the inclusivity space.

Daniel Smith: That’s a really helpful overview. Here in a moment, I want to talk some more about resources that are available to people to help operationalize the practices that you just covered.

But before we talk about that, what are some of the common challenges that researchers or people who are developing new technologies may face when it comes to applying these principles?

Brian Nosek: Yeah. The core challenges are not knowing what to do, not knowing how to do it, not knowing where they can do it, and not necessarily seeing that others are doing it or that they would be rewarded for it.

Not knowing what to do is actually quite common for principles of open science because we haven’t been doing a lot of these behaviors for very long. They’re new concepts to many communities. And actually, bringing them into those communities takes some time. It also takes some investigation of whether these new solutions are actually relevant, and applicable, and helpful for the questions that those research communities are investigating. That requires research about the open science practices themselves, to see if they are fit for purpose.

On how to do it, researchers need help to learn these skills. Just like they learn how to do statistical analysis, or research design, or writing good papers, there needs to be a lot of training and support for adopting these new behaviors that they haven’t done before, and then to learn how to do them well.

For where to do it, there is a growing community of services that support open science behaviors. That’s becoming more visible across more communities, for where they can actually do this research. For example, my organization, the Center for Open Science, hosts the Open Science Framework, which is a free, web-based tool that allows researchers to do all of these open science behaviors. They create projects, they can register their research designs, they share their materials, and their data, and their code. They connect the other types of repositories or services that they used to make more of their research available. And then, they attach their reports and everything at the end. They can make everything freely available for others to interact with.

But to be able to do that and do it well, they need training, and they need to see and get support from others in their community that that effort is worth it. That others are getting value out of the fact that they’re doing the work to be more transparent, to share, to try to be more inclusive in their process.

All of those are contributing and challenges. And then, the last of course, is that the reward system is if the institutions, and the publishers, and the funders don’t realign their incentives to reward researchers for doing these open behaviors, then it’s going to be very hard for researchers who are very busy, who have stress, who work in a competitive environment to prioritize behaviors that aren’t going to help them actually keep their job and thrive in their careers.

Daniel Smith: You mention the Open Science Framework provided by the Center for Open Science as one resource. Can you talk more about that framework and any additional resources that are available to help people overcome these challenges, or just operationalize the practices that you have talked about today?

Brian Nosek: There are a lot of organizations and groups that are providing resources and support for making open science more possible. Many of them are grassroots organizations and groups that have developed online, self-driven tools and resources. There’s one called FORRT, F-O-R-R-T. There’s another called the Turing Way. There’s training and support services that we, the Center, provide ourselves.

But a very simple Google search of open science training will turn up lots of amazing resources for learning one’s self. Getting slides or training tools for one’s classroom. Or actually finding groups that will do trainings on particular behaviors that people are interested in learning about, just conceptually. Or being able to actually translate into practice themselves. On the training side, there’s lots of interesting solutions that are supporting improvement in understanding how to do this.

In terms of doing many of the open science behaviors, the OSF, our infrastructure, the Open Science Framework, is a registry in that it enables preregistration of research designs. There are some other registries that support different areas of research practice. In clinical trials, registration has been a standard, in fact by law, for two decades. Clinicaltrials.gov is a very common registry for people in that domain. But there are emerging resources to help support idiosyncratic needs for registration.

And then the repository community is very big now. There are some generalist repositories, like the OSF and several others. Then there are domain-specific repositories that are worth exploring if one has specific data that has a repository associated with it. Genomic data, for example. Storing that in a genomic database is much more helpful for other researchers to be able to discover it, to be able to join it with other data that is similar, and to be able to use it in interesting ways. This community of repositories is, over time, becoming more aligned with each other so that it’s easier to find the right databases, and then use them to share one’s own research.

Finally, I’ll mention one initiative that’s really been productive for trying to shift the reward system in a way that aligns with researchers’ interests. That is a publishing model called Registered Reports. The idea of Registered Reports is that a journal, instead of doing peer-review on your final report, after you’ve done all of the research, gotten all your findings, written them all up and submitted it to the journal, where the reward there is all about getting the right findings. A Registered Report conducts the primary stage of peer review after the design phase, before you have your results. You send to the journal your plan, “Here’s my research question. Here’s some background information about it. Here’s some preliminary exploratory work about why I’m doing this, or some ideas, or some initial evidence that I might be onto something. Here’s my methodology. Here’s what I’m going to do, and here’s my analysis plan to do it.” The journal does peer review on that. If their reviewers endorse it, then the journal commits to publishing it, regardless of what the outcomes are.

What’s great about this model is that it’s no longer about the results that determine whether the researcher gets their reward, the publication. Instead, it’s about asking important questions and designing really rigorous methods to test those questions. That’s the basis of the reviewers’ decisions to accept the paper for publication or not. That shifts the rewards toward the things that researchers value, and provides a new pathway for researchers to get rewarded in ways that advance openness and rigor of research. At this point, there are more than 350 journals that offer Registered Reports as an option for their publishing.

This, coupled with repositories, coupled with registries, coupled with training, coupled with grassroots groups providing advocacy for these has created a really rich network of open science enthusiasts, open science advocacy, and growth in the community of practice around these new behaviors.

Daniel Smith: That’s wonderful. I certainly encourage everybody to do their own research on some of the available resources out there. But I will include some links in the show notes to the Center for Open Science and the Open Science Framework, because I think those will be great resources for people to get started.

On a final note, what are some emerging trends in open science, or final thoughts you have that we have not already covered?

Brian Nosek: Yeah. I think the things to pay attention to in the open science space are one, it is moving well into the mainstream. 10 years ago, it was a relatively niche activity. But now, open science is recognized, it is valued, and it’s becoming normal behavior across many different disciplines. Different disciplines are at different rates of pursuing and engaging with different behaviors in open science. But it is becoming a widespread phenomenon, suggesting that all communities will be engaged with at least some parts of these practices in the future.

A second thing, in terms of emerging trends, is that the early phase of open science was just getting people to do it at all. Can you please post your data and share that? What has emerged, as a research community evaluating the quality of open science is, “Yeah, we’re doing the behavior, but we’re not really doing it very well yet.” There are lots of limitations to the quality of our shared data, the quality of the preregistrations and planning what’s coming, the quality of being able to actually use the information that has been shared. The next wave of open science is really how do we turn doing the behavior into doing the behavior well? That’s where I think we’ll see lots of really interesting improvements over the next decade.

Daniel Smith: That is a perfect place to leave our conversation for today. Thank you again, Brian.

Brian Nosek: My pleasure. Thanks for having me on.

Daniel Smith: I also invite everyone to visit citiprogram.org to learn more about our courses, webinars, and other podcasts. There are many offerings related to our conversation today, such as courses on scholarly publishing, protocol development, and technology transfer that I encourage you to check out. With that, I look forward you to bringing you all more conversations on all things Tech Ethics.