Introduction to experimental design | High school biology | Khan Academy

What we are going to do in this video is talk a little bit about experiments in science. Experiments are really the heart of all scientific progress. If you think about it, let's just say this represents just baseline knowledge. Then people have hunches in the world, and for a lot of times, people say, "Hey, I have a hunch that that thing is good for you," or "that thing is good for you," but they really had no way of measuring how confident they were. There were no good ways of proving it.

Even more, because they had no good way of proving it, it was hard for people to build on top of that knowledge. But with the scientific method and experiments, people were able to say, "Hey, we have a hypothesis here," and we were able to do some well-designed experiments. So, we feel pretty good that this is true. Then future people are going to say, "Hey, since we feel pretty good that this is true, maybe we can design an experiment to see whether that is true."

"Hey, that actually is true!" Then they can build on that, and we end up having scientific progress that can accumulate over hundreds of years. This is really important because the experiments are well designed. In the future, and this happens all the time, we might realize that, "Hey, actually, there were a few assumptions baked in here that weren't accurate." That allowed us to make essentially misleading conclusions, so our conclusion wasn't quite right.

Then we'll have to rebuild from that point in order to make sure that we are truly making progress. So the key question is, how do we set up well-designed experiments? It's a whole field of study, but the whole point of this video is to really give an introduction to it. So let's just start with a hypothesis. Let's say that you have a hypothesis that some pill, that is made up of the petals of some flower, that this pill right over here improves running speed if someone were to take it.

The important thing of any hypothesis is that it has to be testable. So what you do is you have to think, "Well, how am I going to test it?" Well, what you can do is you can give the pill to some runners and test their time before the pill and after. You might say, "Hey, maybe if I know their times improve after, maybe my hypothesis is correct."

Pause this video and see if you feel comfortable with this test right over here, this experiment. Well, actually, there are several problems with this experiment. How are you selecting these runners? If you give them the pill and their speed improves, did it truly improve because of the pill, or did it improve because of some other thing that they are doing?

Maybe they got new shoes, or maybe their diet improved in some way, or maybe they just had a psychological improvement. This is often known as the placebo effect. If people are taking something that they believe will help them, it often will help them, even if that thing is just an empty capsule or just a sugar tablet.

So how do you avoid these types of errors? Well, what you could do is you can find runners and put them into two groups. Let's say this is one group right over here, and then this is another group. What you would want to do is go into the population of people and randomly select whether someone goes into one group or another group. Why random? Because if you don't randomly select, there's a chance that there might be some implicit bias.

You might just happen to be picking people whose running speed is on an upward trajectory, and they just happen to go into the group that will eventually get the pill. So you randomly put them in those groups. What you want to do is have a control group and a group that gets your pill. Now, you might be tempted for this group to say, "Well, they don't get a pill."

Then after a few months of it, and it should be the same amount of time, you ask, "Hey, did this group's times improve over the 100 meters? How did that compare to this group?" But be very careful. If this group gets the pill and this group gets nothing, then the pill might be providing that placebo effect again. Just making people think they're getting something that's making them faster might actually be a self-fulfilling prophecy.

So, it's important that you also give these people a pill, although this pill would just look like a pill. This would be just an empty pill that looks the same. Now there’s another idea when you’re designing scientific experiments that it needs to be double-blind. Let me write this down: double-blind.

As you can imagine, it implies that two things are blind here. The first thing that needs to be blind is the people themselves should not know which group they're getting put into. They should not know which pill they're taking. Because if you put someone in this group and say, "Hey, you're in the control group; we're just going to give you an empty pill," well, then the placebo effect might not be effective.

It's also important, because it's double-blind, that the people who are working with the runners—the people who are measuring them, the researchers—do not know which group they are administering it to. Because if they did, they might be able to signal somehow or even subconsciously give a sense of which group folks are in.

So let’s say we do all of these things. We’re getting in the direction of a well-designed experiment, and we find that the 10 people in this group versus the 10 people in this group, after three months, these folks had a 5% improvement in running speed, and these people had a 10% improvement in running speed. Is that enough to conclude that our hypothesis is correct?

You might be tempted; it seems suggestive. But that's where statistics come into order. Because there’s just some random chance that you got lucky—you might have just happened to pick people whose running speed was improving more, and maybe your pill does nothing.

There’s a whole field of inferential statistics. When you take a statistics course, it will go in more depth into this. Essentially, what you're going to do is say, "Hey, assume that your pill does nothing. What's the probability of getting this result for 10 people or what's the probability of getting this difference in result?" If that probability is very low, well, you would say, "Hey, that would suggest that my pill actually does do something."

Now another important principle of an experiment like this is it needs to be replicable. Because even though you thought you did a good job, people might not want to take your word for it. It's important in science for people to be skeptical. When people do experiments, they want to have a result, and that bias might creep in.

If someone else does an experiment, you need to say how you did that experiment so other people can see if they get the same results. Because even though you think you randomly selected, you might only do it with people from a certain country or under certain weather conditions. These people might do it slightly differently or in a different country or under different constraints, and realize that, "Hey, the explanation for this maybe was something else."

Another thing to keep in mind is the larger your samples, the larger groups that you're able to do this with, the stronger that the statistics actually become. I would say not just larger, but the more diverse—across genders, across ethnicities, across geographies.

The big picture here is that all scientific progress is based on us designing good experiments and being very rigorous about how we think about those experiments. What I've highlighted here is just the beginning of how we might think about designing those experiments. As you go into your scientific careers, look at other people's experiments and see whether they've done these things, because many times you will find that it is not as rigorous as it might seem at first.