Critical thinking about evidence

My name is Dr. Cynthia Nebel, and I'm the director of learning services at St. Louis University School of Medicine, and today I'm going to talk to you about critical thinking about evidence.

A lot of times we might hear that some particular finding or strategy is evidence-based. What exactly does that mean? It turns out that there are a lot of different kinds of evidence. I'm going to walk you through a few different kinds of evidence that might be used in different situations, so that you can better think about what evidence is being used in the situation that you are given, so that you can think about whether or not you actually buy what it is they are trying to sell. The first is something called corporate sponsored research. Now, it turns out that in our world there are lots of products that individuals will try to sell us, and a lot of that has to involve some amount of research to make sure that the product works well. Corporate-sponsored research is not necessarily a bad thing. However, it does mean that the researcher who has done that research has received money from that organization with the primary purpose of trying to show that particular product works the way that they say it does. This does not mean that the research is necessarily bad, but it does mean that we need to be a little bit skeptical. We need to look into what kind of methods that particular research used, so that we could make sure that they did everything appropriately and left open the possibility that the product might not work. The next types of evidence are case studies or anecdotes. These are used very, very often, especially in education research where individuals might try something in their classroom, see that it worked and write it up and publish that. That could be really useful for us to learn about some idea that someone had that worked well for them. The problem with case studies and anecdotes is that just because it worked for one person or one group of people doesn't necessarily mean that it'll work for others. For example, we know that there are some strategies that work really well for the neurotypical population but might not work as well for the neurodivergent population. We know that there are some strategies that work well for students who come from a particular background and not as well for others. Or we know that there are some strategies that work great for certain age groups, but we can't do the same thing with young children as we do with college students. So a case study or an anecdote is a great way of producing new ideas, but it's not the best evidence for whether or not something is going to work in a new situation. The next evidence that is often used is correlational. Correlations are where research is conducted and individuals are looking for relationships between variables. As the speed of running goes up, so does the heart rate of the individual. There's a relationship between those two variables. We could look at other kinds of variables as well. Lots of different things involve correlational research, and lots of things have to. For example, we might look at blood flow to a particular area of the brain to correlate it with the number of times someone laughs in a situation. So we're looking at humor associated with brain activity. In fact, most studies of brain activity are correlational. Why? Because we can't actually go in and turn on and off different parts of the brain to say that they work differently. So correlational research is really prevalent. Most survey research is correlational in nature, where we ask individuals questions, and we say, "Ah, the people who said this are also more likely to say this." All of that is correlational, and it's really good research. But there is one very big limitation with correlational work, and that is we can't draw causal conclusions. So take one very common correlational finding. Happy kids tend to get better grades. That's true. There's a very strong correlation between happiness or satisfaction in children and their academic performance. When we really start to dig into that, we can't necessarily talk about this in a causal way. Is it true that happiness leads to higher academic performance? Possibly. That is one possibility that if we make kids happy, they will perform better. It's also possible that high performance leads to more satisfaction. The kids who are performing well tend to also be happier because they're performing well. There's also a third possibility that these two things aren't actually related to each other at all. But home life affects both. When you've got a happy home life that is stable with parents who help you out, that leads to higher happiness scores and to higher academic performance. Even though correlations can be extremely compelling, we have to be very careful about the way that we interpret that data. The last type of evidence is called a true experiment. An experiment is when we randomly assign people to two different groups and we try to only manipulate one thing, one variable, and hold everything else constant. Now, experiments can be a little bit tricky to understand, so I'd like to give you some examples of a true experiment so that you can see this a little bit better.