Worked example identifying experiment | Study design | AP Statistics | Khan Academy

So we have some type of study described here. I encourage you to pause this video and think about whether this is an observational study, an experiment, or a sample study. Also, think about whether it was conducted well.

All right, now let's do this together. A group of doctors was interested in comparing the effectiveness of placebo pills and real pills in treating migraines. Placebo pills are pills that look just like the regular pill, and from a patient's point of view, they don't know which one they're getting. The reason why we do this in studies is because there's something called the placebo effect. Oftentimes, just by taking something that you think is good for you, some medicine that you think might help you, it actually does help you. We can think about why the placebo effect happens, but this is well documented.

When people are trying to test medicine, they want to say, "Well, does this have an effect above and beyond the placebo effect?" That's why they are putting them in these two groups. That's why they're comparing the real pills to the placebo pills.

They randomly assigned a group of 300 patients suffering from migraines into two groups. So they have their 300 patients right over here, and they're randomly putting them into the placebo group or the real pill group. One group was given a real pill. Oh, we already… we already went… we already went read that. Both groups were told which kind of pill they got. That is sketchy because the whole point about a placebo is that you think you got the real thing, or you think you might have gotten the real thing. If you're told you got a placebo, that tends to undermine the placebo effect. If you're just told you're given a sugar pill, well then the impact of—well, it's well anyway. This is definitely bad practice.

So let's keep going here. Before taking the pills and a day afterwards, the patients were asked to fill out questionnaires regarding their condition. Then the doctors analyzed the overall changes in questionnaires for each group and compared them.

All right, so first of all, what type of study is this? Well, we're taking our groups; we are randomly putting them into two different groups. You could call the placebo group maybe the control group, and then you're putting the real pill as your actual experimental group. This is a classic experiment. You're trying to establish a causal relationship; you want to see whether this real pill actually makes migraines better—my migraines better. So, does it actually do it, and does it actually do it better than a placebo? You're randomly putting the people in both groups to try to distribute any confounding variables that there might be.

So this is clearly an experiment. Now the other options you might say, "Well, is this maybe an observational study?" Remember, in an observational study, that's more of where you look at a population, you look at a group, you ask them a bunch of questions often or you make a bunch of observations and you see if there's correlations between two variables—so variable one and variable two. You're able to make some type of correlational statement, and you're not trying to get at causality.

In a sample study, this is just you trying to estimate a parameter for the entire population. A sample study might have been, um, of the entire population, what percentage gets migraines? You can't talk to the entire population; maybe the entire population is millions of people. So you take a sample of, say, 100 people and you ask them, "Do you get migraines?" Then you say, "Okay, that percentage of our sample that gets migraines, that's a good estimate for the parameter of what percentage of my entire population actually gets migraines."

So this was clearly an experiment. Now the next question is, "Well, was this experiment conducted well?" Well, even when I read it, I was bothered by both groups being told which kind of pill they got. That completely defeats the purpose of a placebo. The placebo effect is, "Hey, I think I'm taking something good for me." It's been documented that when you think you're taking a pill that helps you, it oftentimes does help you.

So if someone's coming up with a new medicine, it better be better; it better perform better, or it better be more effective than just that placebo. I don't like the fact—this was a very bad study to tell both groups what kind of pill they got. You actually should tell neither group which type of pill they got.