Tutorial 7: Statistical Hypothesis Testing

The principles of statistical hypothesis testing

Week of October 24, 2022
(7th week of classes)

How tutorials work

The DEADLINE for your report is always at the end of the tutorial.

The INSTRUCTIONS for this report is found at the end of the tutorial.

Students may be able after a while to complete their reports without attending the synchronous lab/tutorial sessions. That said, we highly recommend that you attend the tutorials as your TA is not responsible for assisting you with lab reports outside of the lab synchronous sections.

The REPORT INSTRUCTIONS (what you need to do to get the marks for this report) is found at the end of this tutorial.

Your TAs

Section 101 (Wednesday): 10:15-13:00 - John Williams ()
Section 102 (Wednesday): 13:15-16:00 - Hammed Akande ()
Section 103 (Friday): 10:15-13:00 - Michael Paulauskas ()
Section 104 (Friday): 13:15-16:00 - Alexandra Engler ()

General Information

Please read all the text; don’t skip anything. Tutorials are based on R exercises that are intended to make you practice running statistical analyses and interpret statistical analyses and results.

Note: Once you get used to R, things get much easier. Try to remember to:

  1. set the working directory
  2. create a RStudio file
  3. enter commands as you work through the tutorial
  4. save the file (from time to time to avoid loosing it) and
  5. run the commands in the file to make sure they work.
  6. if you want to comment the code, make sure to start text with: # e.g., # this code is to calculate…

General setup for the tutorial
A good way to work through is to have the tutorial opened in our WebBook and RStudio opened beside each other.

The principles of statistical hypothesis testing using a simple problem: do toads exhibit handedness?

The right hand of toads (as seen in class, but a small summary is provided here)

Statistical hypothesis testing is the most used statistical framework to generate evidence for or against questions based on data (research questions or otherwise).

From Whitlock and Schluter (2014): “Humans are predominantly right handed. Do other animals exhibit handedness as well? Bisazza et al. (1996) tested this hypothesis on the common toad. They sampled (randomly) 18 toads from the wild. They wrapped a balloon around each individual’s head and recorded which forelimb each toad used to try to remove the balloon. Do other animals exhibit handedness as well?”

First step: transform the research question intoto a statistical question

Statistically, the question above can be translated into: “Do right-handed and left-handed toads occur with equal frequency in the toad population, or is one type more frequent than the other?”

Result of the study: 14 toads were right handed and four were left handed. Are these results evidence of handedness in toads?

Do the results provide evidence to say that the sample value we obtained is consistent or inconsistent with the samples from a “theoretical” population of toads where right- and left-handed individuals are equal in proportion? If inconsistent, then the data would have generated evidence that other animals exhibit handedness. This part of the tutorial (below) was adapted from Whitlock and Schluter’s code in https://rpubs.com/mdlama/153914.

Since we sampled 18 toads, we will use this sample size to produce our sampling distribution for the theoretical population in which there is no dominance of one limb (hands) over the other limb.

Let’s start by learning how to take a random sample from a categorical variable having two possible outcomes from a theoretical population of known probabilities (here, 50%/50% chance of each hand) with the desired sample size (i.e., 18 individuals).

The function below will simulate the following “manual” implementation:

  1. put millions of small pieces of papers into a bag where half are written “Left” and the other half “Right”. In practice, if we replace the sample piece of paper back into the bag each time we sample, we only need 36 pieces so by chance alone zero “Left” and 18 “Right” or any combination would be possible. See point #2 below for more explanation.

  2. Sample with replacement 18 pieces of paper from the bad. Replacement means that you will take one piece of the paper, write down whether you got a “L” or “R” and then put back in the bag before taking another piece of paper. Replacement here is critical because otherwise you would change the original theoretical population while sampling it. If you were to keep the piece of paper (i.e., sampling without replacement), you would reduce the probability of sampling “Ls” and “Rs”. For example, imagine that your first 3 values were “Ls” and that you did not replace them. Then, you increased the probability of sampling an “R”. Remember one of the criterion for random sampling “The selection of observational units in the population (e.g., individual piece of paper here) must be independent, i.e., the selection of any unit (e.g., L or R) of the population must not influence the selection of any other unit.”

The function below is pretty self explanatory. But some details are perhaps useful. c("L", "R") sets the categories that we require; we could have written “Left” and “Right” but “L” and “R” makes the code look “simpler”. If three categories were required, we could had set them as c("A", "B","C"). size is the sample size. prob is the probability of each category. Let’s say we had 3 categories with 33.33% each as probability. Then prob = c(0.3333, 0.3333,0.333). Lastly replace = TRUE states (for the purpose of the problem here) that each observation (“toad”) has the same probability of being sample.

Sample1 <- sample(c("L", "R"), size = 18, prob = c(0.5, 0.5), replace = TRUE)
Sample1

We can then calculate how many individuals in sample 1 were of a particular category (here we decided right-handed); you will see later that selecting the left-handed wouldn’t make a difference as we know how many there are (i.e., 18 individuals):

sum(Sample1 == "R")

Let’s take a second sample and calculate its sum:

Sample2 <- sample(c("L", "R"), size = 18, prob = c(0.5, 0.5), replace = TRUE)
Sample2
sum(Sample2 == "R")

Most likely Sample1 and Sample2 have different numbers of right-handed individuals.

Now, let’s generate a huge number of samples from the theoretical population (i.e., computer approximation of the infinite sampling process done via calculus) and for each of them keep the score of right-handed toads:

number.samples <- 100000
samples.n18 <- replicate(number.samples,sample(c("L", "R"), size = 18, prob = c(0.5, 0.5), replace = TRUE))
dim(samples.n18)

Let’s print the first 3 samples (remember that each sample is in a different column of the matrix samples.n18:

samples.n18[,1:3]

Let’s now calculate for each sample, the number of individuals that are right-handed:

results.n18 <- apply(samples.n18,MARGIN=2,FUN=function(x) sum(x == "R"))

The number of left-handed toads in each of the 100000 samples can be calculated simply as:

18-results.n18

Take a look into what the vector results.n18. Each value in the vector results.n18 represents the number of right-handed toads from a random sample of 18 individuals from a theoretical population in which the number of right- and left-handed toads are the same. Remember the function head lists the first 6 values in a vector or matrix:

head(results.n18)

Let’s build the frequency distribution table for the samples (i.e., sampling distribution of right-handed toads from the theoretical population):

Table.TheoreticalPop <- table(results.n18, dnn = "numberRightToads")
Table.TheoreticalPop

Note that you may end up with no sample with all 18 individuals being right-handed or no individual being right-handed (i.e., all 18 being left-handed). Remember that we took 100000 samples and this may be not enough to generate all possible types of outcomes. When I did 1000000 (1 million), then I get them. In reality, we use much larger number of samples for a computational approach; and when using an analytically-based solution (calculus-based), then the distribution contains (approximates) infinite number of samples. However, our attempt here is provide you the knowledge and intution underlying statistical hypothesis testing. And the computational approach even based on just 100000 values allows to build this knowledge; and actually approximates the true distribution very closely.

Obviously the number of samples in Table.TheoreticalPop is 100000:

sum(Table.TheoreticalPop)

We can also calculate the relative frequency of samples of different number of right-handed toads as:

Table.TheoreticalPop/100000

As you could have predicted, the most common samples are those in which half of them are right- and the other half left-handed. But notice that many samples differed from this value by chance alone. This is simply due to sampling variation around the theoretical value (50%/50% right- and left-handed). To make the table to look a bit better, let’s use the function data.frame:

data.frame(Table.TheoreticalPop)

We can also calculate the probability of each class (i.e., for each number of individuals that are right-handed) as follows:

Table.TheoreticalPop/100000

Because we are dealing with a discrete variable (number of right-handed toads), the histogram will lump discrete values (number of right-handed toads) into classes. In this case, we rather use a barplot using the table we generated for the theoretical population so that each discrete value appears separately:

barplot(height = Table.TheoreticalPop, space = 0, las = 1, cex.names = 0.8, col = "firebrick", xlab = "Number of right-handed toads", ylab = "Frequency")

The distribution is symmetric but note that the relative frequency 8 left- handed toads may not be exactly the same as 10-right handed toads; and the same is the case for the relative frequencies of 2-right handed and 16-right handed toads. This is because we did not generate the distribution for infinite number of samples; note, however, that they are pretty similar!

Now, let’s go back to our original data (i.e., real sample) in which 14 toads were right-handed. What is the the probability of obtaining a sample as extreme or more extreme than 14 toads?

frac14orMore <- sum(results.n18 >= 14)/number.samples
frac14orMore

Given that the sampling distribution is very symmetric, the value above is pretty similar to:

frac4orLess <- sum(results.n18 <= 4)/number.samples
frac4orLess

They differ slightly because we used a computational approach. If infinite samples were taken, then frac14orMore and frac4orLess would had been the same because the sampling distribution would have been perfectly symmetric. That’s the reason we counted one limb (right), i.e., because the counts for left-handed toads can be easily calculated by subtraction.

Given that the interest here is not to generate evidence that the dominant limb of toads is their right (or left) limb but rather generate evidence as to whether they have a dominant limb (either right or left), we adopt a probability that reflects both sides of the sampling distribution from the theoretical population:

2*frac14orMore

This probability can be equally estimated as:

frac4orLess+frac14orMore

Again, note that the difference between the two values above are due to the fact that our distribution was computationally generated. The one generated by infinite sampling would had given exactly the same value, i.e., frac4orLess+frac14orMore would had been exactly the same as 2*frac14orMore.

The computational approach developed here is meant to facilitate students to understand the principles involving generating statistical evidence for or against a scientific hypothesis. In real applications, however, we would had used a method that generates the sampling distribution based on infinite sampling (referred as to an exact test); and not computationally as we did here. The probability based on an exact test can be estimated as follows:

binom.test(14,18,(1/2),alternative="two.sided")$p.value

The probability is 0.03088379, which is pretty close to our computational approach. In my case, 2 x frac14orMore was equal to 0.03194.

Regardless of the approach (infinite or computational), the probability of finding a sample of 18 toads (from a theoretical population in which the number of right- and left-handed individuals are equal) in which 14 are right-handed and 4 are left-handed (hence calculating the probabilities from both sides of the curve) is around 0.03. This probability tells us how unusual is to find a sample like the one found by the study (i.e., 14 right-handed and 4 left-handed) in a theoretical population. The probability is quite small (0.03111), thus providing evidence to state that the toads in the real data appear to exhibit handedness. In other words, the probability of finding a sample value with 14 toads as right-handed (4 left-handed) is quite implausible (inconsistent) with what is expected from a theoretical population with half of the individuals being right-handed and the other half being left-handed!

As such, our original sample of 14 right-handed toads and 4 left-handed toads seem to come from a population different from our theoretical population of 50% right-handed and 50% left-handed. We then use this probability to generalize the results and state that “the study generated evidence that toads present handedness.

Report instructions
All the information and code needed for you to solve this problem is in the tutorial.

The report is based on the following problem adapted from Wikipedia (https://en.wikipedia.org/wiki/Ideal_free_distribution): The ideal free distribution (IFD; Fretwell and Lucas 1970) is a theory in ecology that states that the number of individuals from a given species that will aggregate in various patches is proportional to the amount of resources available in each patch. For example, if patch A contains twice as many resources as patch B, there will be twice as many individuals foraging in patch A as in patch B. The IFD theory predicts that the distribution of individuals among patches will minimize resource competition and maximize fitness.

Let’s assume that you run an experiment to test the IFD in a species of grasshoppers. You set two patches (two tanks linked by a tube) and put 50% of the food in one tank and 50% of the food in the other tank; you then place 20 grasshoppers in one tank and 20 grasshoppers in the other tank. Then you go back after 6 hours and count the number of grasshoppers on each tank (individuals may have moved between tanks). The result is that 6 individuals were found in tank 1 (and obviously 34 individuals were found in tank 2, i.e., no individual died during the experiment).

The report also needs to include the answer to the following question: Do these results (probability value) provide evidence for or against the Ideal Free Distribution?

Develop code to estimate whether the results are consistent or not with the IDF. Code should include both the computational and the exact test. Report results by writing a few sentences interpreting the results in light of the problem and the probability (likelihood) of the sample result assuming that the IDF is true (i.e, distribution of individuals is proportional to the amount of resources). Text for your report (not code), as before, should be reported as:

The RStudio file name should be: Tutorial7YourLastName

Write some text explaining your answer on the basis of the code and the concepts you have learned above.

# use as many lines as necessary,
# comments always starting with  #

Reference: Fretwell, S. D. & Lucas, H. L., Jr. 1970. On territorial behavior and other factors influencing habitat distribution in birds. I. Theoretical Development. Acta Biotheoretica 19: 16–36.
Submit the RStudio file via Moodle.