Part 1D: Hypothesis Testing and p-values

The above activity can be used to explain the core ideas behind all statistical hypothesis tests. Hypothesis testing is a process used to determine whether an event can reasonably be attributed to chance or whether there is some other explanation.

For example, let’s assume your friend, Akilah, claims to be psychic and you decide to test this claim using a test similar to the one above. There are two possible conclusions we can make.

Based on the data visualization in Part 1C, we can make conclusions based on Akilah’s score.

If Akilah played your game and got 4 out of 10 correct, she would have done better than expected. However, the theoretical distribution in the above app shows us that 12% of the time people can get 4 or more correct just by random chance. In terms of a statistical hypothesis test, the p-value = 0.12 does not give us enough evidence to reject Claim 1.

A p-value is a number between 0 and 1 that we use to quantify our decision. A p-value is the probability of observing an outcome assuming that the null hypothesis is true. When a p-value is very small, it means it is very unlikely that the null hypothesis is true.

For example, if the null hypothesis is true (p = 0.2) the probability that Akilah correctly guesses 6 or more cards is 0.01. This may cause us to question the null hypothesis and conclude that something is helping Akilah correctly guess the cards.

However, a p-value never proves that our alternative hypothesis is true, it simply tells us how unlikely the null hypothesis is when only random chance is involved. Remember that a p-value = 0.01 means that 1 out of 100 times someone will correctly guess 6 or more cards just by random chance even when the null hypothesis is true.

Part 1E: Get Curious

1. How did your class do? Sometimes it can take up to 60 minutes for your class data to show in the App above. After your instructor confirms enough class data is available, answer the following questions.

a) Select your GroupID and use the summary statistics button to identify the mean, median and sample size of your class data.
b) Compare the histogram of your class data to the theoretical binomial distribution. Is the center, spread, and shape similar?
c) If each student in your class played the game 10 times, providing 10 times more results, would you expect the class data to look more or less like the theoretical binomial distribution?
d) What was the best score in your class? If there are 20 people in your class (20 games played), would you be surprised if one person got a “psychic” result by getting 5 or more correct?

2. How does the number of cards influence our results? Play the game one more time with 10 attempts, however, only select 2 cards per attempt instead of 5 cards per attempt.
a) In the first game the probability of success was 1/5 = 0.2. What is the probability of success when there are only 2 cards per attempt?
b) When you have 10 attempts, with 2 cards each, how many successes would you expect to get?
would you expect to get? c) Use the App above to determine how many successes would be needed before you consider the result to be unusual

3. In 2019, USA Today, stated that “For the 12th year in a row, Mote Marine Laboratory and Aquarium's manatees, brothers Hugh and Buffett, predicted the winner for the Super Bowl.” Watch the video here: https://www.youtube.com/watch?v=KDUet_DD6q4 and you can see that Hugh has correctly predicted the Super Bowl 9 out of 11 times.

a) In this example, we can consider each year as an attempt and instead of using cards, they are selecting teams. For any given year, what is the probability that Hugh and Buffet correctly guess the winner?
b) Use the above App to estimate how likely is to get 9 out of 10 attempts correct? Does this cause you to think something other than random chance is involved?
c) Hugh had correctly picked 9 winners from 2008-2018 while Buffet had only correctly picked 6 winners. Does this mean that you should have gone with Hugh’s 2019 pick?
d) The USA Today article also shows other animals that have attempted to guess the Super Bowl winners, such as Fiona the Hippo, Kiki the Lioness, April the Giraffe, and many others. Explain why it is NOT unusual for one animal to be able to predict so well.

4. Your friend Eli, claims that he can tell the difference between two carbonated beverages, Coke and Pepsi. However, you are skeptical and think they taste the same. So you design a test to determine if Eli is right. You secretly pour Coke into one cup and Pepsi into a second cup. You give both cups to Eli at the same time and have him tell you which is Pepsi. You repeat this process 10 times and record the number of guesses that Eli got correct.
a) State the null and alternative hypotheses.
b) If the null hypothesis is true, would you be surprised if Eli got 7 out of 10 correct? Use the App to explain your reasoning
c) In this test, Eli got 9 out of 10 correct. Use the App to find the p-value for this test. Based on this p-value would you conclude that the null hypothesis is correct (and Eli was just lucky) or would you conclude that the alternative is true (Eli can taste the difference between the two)? Explain your reasoning.

5. How does the number of attempts (or sample size) influence our conclusions? Use the App to find the following probabilities for the psychic game.
a) With 5 cards and 10 attempts, how likely is it that a person will get 40% correct (4 out of 10 correct guesses)?
b) With 5 cards and 20 attempts, how likely is it that a person will get 40% correct (8 out of 20 correct guesses)?
c) With 5 cards and 50 attempts, how likely is it that a person will get 40% correct (20 out of 50 correct guesses)?
d) Explain the pattern you see from part 5a – 5c. Why would you expect a larger number of observations to influence the probability of an outcome?

References

Shannon, Joel, Feb 2019, Adorable animals across the nation are making Super Bowl predictions, USA Today. https://www.usatoday.com/story/news/nation/2019/02/03/animals-predict-super-bowl-outcome/2756507002/