---
title: "Introduction to Statistics in R"
author: "Joschka Schwarz"
---

```{r}
#| include: false
source(here::here("R/setup-ggplot2-tie.R"))
options(dplyr.summarise.inform = FALSE)
```

Statistics is the study of how to collect, analyze, and draw conclusions from data. It’s a hugely valuable tool that you can use to bring the future into focus and infer the answer to tons of questions. For example, what is the likelihood of someone purchasing your product, how many calls will your support team receive, and how many jeans sizes should you manufacture to fit 95% of the population? In this course, you'll use sales data to discover how to answer questions like these as you grow your statistical skills and learn how to calculate averages, use scatterplots to show the relationship between numeric values, and calculate correlation. You'll also tackle probability, the backbone of statistical reasoning, and learn how to conduct a well-designed study to draw your own conclusions from data.

# 1. Summary Statistics

Summary statistics gives you the tools you need to boil down massive datasets to reveal the highlights. In this chapter, you'll explore summary statistics including mean, median, and standard deviation, and learn how to accurately interpret them. You'll also develop your critical thinking skills, allowing you to choose the best summary statistics for your data.

## What is statistics?

Theory. Coming soon ...


**1. What is statistics?**

Hi and welcome to the course. My name is Maggie, and I'll be your host as we dive in to the world of statistics.

**2. What is statistics?**

So what is statistics anyway? We can talk about the field of statistics, which is the practice and study of collecting and analyzing data. We can also talk about a summary statistic, which is a fact about or summary of some data, like an average or a count.

**3. What can statistics do?**

A more important question, however, is what can statistics do? With the power of statistics, we can answer tons of different questions like:How likely is someone to purchase a product? Are people more likely to purchase it if they can use a different payment system?How many occupants will your hotel have? How can you optimize occupancy?How many sizes of jeans need to be manufactured so they can fit 95% of the population? Should the same number of each size be produced?A question like, Which ad is more effective in getting people to purchase a product? can be answered with A/B testing.

**4. What can't statistics do?**

While statistics can answer a lot of questions, it's important to note that statistics can't answer every question. If we want to know why the TV series Game of Thrones is so popular, we could ask everyone why they like it, but they may lie or leave out reasons. We can see if series with more violent scenes attract more viewers, but even if they do, we can't know if the violence in Game of Thrones is the reason for its popularity, or if other factors are driving its popularity and it just happens to be violent.

**5. Types of statistics**

There are 2 main branches of statistics: descriptive statistics and inferential statistics. Descriptive statistics focuses on describing and summarizing the data at hand. After asking four friends how they get to work, we can see that 50% of them drive to work, 25% ride the bus, and 25% bike. These are examples of descriptive statistics.Inferential statistics uses the data at hand, which is called sample data, to make inferences about a larger population. We could use inferential statistics to figure out what percent of people drive to work based on our sample data.

**6. Types of data**

There are two main types of data. Numeric, or quantitative data is made up of numeric values. Categorical, or qualitative data is made up of values that belong to distinct groups.It's important to note that these aren't the only two types of data that exist - there are others too, but we'll be focusing on these two.Numeric data can be further separated into continuous and discrete data. Continuous numeric data is often quantities that can be measured, like speed or time. Discrete numeric data is usually count data, like number of pets or number of packages shipped.Categorical data can be nominal or ordinal. Nominal categorical data is made up of categories with no inherent ordering, like marriage status or country of residence. Ordinal categorical data has an inherent order, like a survey question where you need to indicate the degree to which you agree with a statement.

**7. Categorical data can be represented as numbers**

Sometimes, categorical variables are represented using numbers. Married and unmarried can be represented using 1 and 0, or an agreement scale could be represented with numbers 1 through 5. However, it's important to note that this doesn't necessarily make them numeric variables.

**8. Why does data type matter?**

Being able to identify data types is important since the type of data you're working with will dictate what kinds of summary statistics and visualizations make sense for your data, so this is an important skill to master.For numerical data, we can use summary statistics like mean, and plots like scatterplots, but these don't make a ton of sense for categorical data.

**9. Why does data type matter?**

Similarly, things like counts and barplots don't make much sense for numeric data.

**10. Let's practice!**

Time to master these important skills!

## Descriptive and inferential statistics

Statistics can be used to answer lots of different types of questions, but being able to identify which type of statistics is needed is essential to drawing accurate conclusions. In this exercise, you'll sharpen your skills by identifying which type is needed to answer each question.

Identify which questions can be answered with descriptive statistics and which questions can be answered with inferential statistics.

| Descriptive | Inferential |
| --- | --- |
| Given data on all 100,000 people who viewed an ad, what percent of people clicked on it? | Given data on 20 fish caught in a lake, what's the average weight of all fish in the lake? |
| Given data on every customer service request made, what's the average time it took to respond? | After interviewing 100 customers, what percent of *all* your customers are satisfied with your product? |

## Data type classification

In the video, you learned about two main types of data: numeric and categorical. Numeric variables can be classified as either discrete or continuous, and categorical variables can be classified as either nominal or ordinal. These characteristics of a variable determine which ways of summarizing your data will work best.

Map each variable to its data type by dragging and dropping.

| Continuous numeric | Discrete numeric | Categorical |
| --- | --- | --- |
| Kilowatts of electricity used | Number of items in stock | Brand of a product |
| Air temperature | Number of DataCamp courses taken | Postal Code |
| | Number of clicks on an ad | |

## Measures of center

Theory. Coming soon ...


**1. Measures of center**

In this lesson, we'll begin to discuss summary statistics, some of which you may already be familiar with, like mean and median.

**2. Mammal sleep data**

In this video, we'll look at data about different mammals' sleep habits.

**3. Histograms**

Before we dive in, let's remind ourselves how histograms work. A histogram takes a bunch of data points and separates them into bins, or ranges of values. Here, there's a bin for 0 to 2 hours, 2 to 4 hours, and so on. The heights of the bars represent the number of data points that fall into that bin, so there's one mammal in the dataset that sleeps between 0 and 2 hours, and nine mammals that sleep two to four hours. Histograms are a great way to visually summarize the data, but we can use numerical summary statistics to summarize even further.

**4. How long do mammals in this dataset typically sleep?**

One way we could summarize the data is by answering the question, How long do mammals in this dataset typically sleep? To answer this, we need to figure out what the "typical" or "center" value of the data is. We'll discuss three different definitions, or measures, of center: mean, median, and mode.

**5. Measures of center: mean**

The mean, often called the average, is one of the most common ways of summarizing data. To calculate mean, we add up all the numbers of interest and divide by the total number of data points, which is 83 here. This gives us 10-point-43 hours of sleep. In R, we can use the mean function, passing it the variable of interest.

**6. Measures of center: median**

Another measure of center is the median. The median is the value where 50% of the data is lower than it, and 50% of the data is higher. We can calculate this by sorting all the data points and taking the middle one, which would be index 42 in this case. This gives us a median of 10.1 hours of sleep. In R, we can use the median function to do the calculations for us.

**7. Measures of center: mode**

The mode is the most frequent value in the data. If we count how many occurrences there are of each sleep_total and sort in descending order, there are 4 mammals that sleep for 12.5 hours, so this is the mode.The mode of the vore variable, which indicates the animal's diet, is herbivore. Mode is often used for categorical variables, since categorical variables can be unordered and often don't have an inherent numerical representation.

**8. Adding an outlier**

Now that we have lots of ways to measure center, how do we know which one to use? Let's look at an example. Here, we have all of the insectivores in the dataset.

**9. Adding an outlier**

We get a mean sleep time of 16.5 hours and a median sleep time of 18.9 hours.

**10. Adding an outlier**

Now let's say we've discovered a new mystery insectivore that never sleeps.

**11. Adding an outlier**

If we take the mean and median again, we get different results. The mean went down by more than 3 hours, while the median changed by less than an hour. This is because the mean is much more sensitive to extreme values than the median.

**12. Which measure to use?**

Since the mean is more sensitive to extreme values, it works better for symmetrical data like this. Notice that the mean, in red, and median, in blue, are quite close.

**13. Skew**

However, if the data is skewed, meaning it's not symmetrical, like this, median is usually better to use. In this histogram, the data is piled up on the right.

**14. Skew**

Data that looks like this is called left-skewed data.

**15. Skew**

When data is piled up on the left, it's right-skewed.

**16. Which measure to use?**

When data is skewed, the mean and median are different. The mean is pulled in the direction of the skew, so it's lower than the median on the left-skewed data, and higher than the median on the right-skewed data. Because the mean is pulled around by the extreme values, it's better to use the median since it's less affected by outliers.

**17. Let's practice!**

Let's practice using measures of center.

## Mean and median

In this chapter, you'll be working with the <a href="https://www.nu3.de/blogs/nutrition/food-carbon-footprint-index-2018">2018 Food Carbon Footprint Index</a> from nu3. The `food_consumption` dataset contains information about the kilograms of food consumed per person per year in each country in each food category (`consumption`) as well as information about the carbon footprint of that food category (`co2_emissions`) measured in kilograms of carbon dioxide, or CO2, per person per year in each country.

In this exercise, you'll compute measures of center to compare food consumption in the US and Belgium using your `dplyr` skills. 

**Steps**

1. Create two data frames: one that holds the rows of `food_consumption` for `"Belgium"` and the another that holds rows for `"USA"`. Call these `belgium_consumption` and `usa_consumption`.

```{r}
# Load packages
library(dplyr)

food_consumption <- readRDS("data/food_consumption.rds")

# 1.1 Filter for Belgium
belgium_consumption <- food_consumption |> 
  filter(country == "Belgium")

# 1.2 Filter for USA
usa_consumption <- food_consumption  |> 
  filter(country == "USA")
```

2. Calculate the mean and median of kilograms of food consumed per person per year for both countries.

```{r}
# 2.1 Calculate mean and median consumption in Belgium
mean(belgium_consumption$consumption)
median(belgium_consumption$consumption)
```

```{r}
# 2.2 Calculate mean and median consumption in USA
mean(usa_consumption$consumption)
median(usa_consumption$consumption)
```

3. Filter `food_consumption` for rows with data about Belgium and the USA.
4. Group the filtered data by `country`.
5. Calculate the mean and median of the kilograms of food consumed per person per year in each country. Call these columns `mean_consumption` and `median_consumption`.

```{r}
food_consumption  |> 
  # 3. Filter for Belgium and USA
  filter(country %in% c("Belgium", "USA"))  |> 
  # 4. Group by country
  group_by(country)  |> 
  # 5. Get mean_consumption and median_consumption
  summarise(mean_consumption   = mean(consumption),
            median_consumption = median(consumption))
```

Marvelous mean and median calculation! When you want to compare summary statistics between groups, it's much easier to do a `group_by()` and one `summarize()` instead of filtering and calling the same functions multiple times.

## Mean vs. median

In the video, you learned that the mean is the sum of all the data points divided by the total number of data points, and the median is the middle value of the dataset where 50% of the data is less than the median, and 50% of the data is greater than the median. In this exercise, you'll compare these two measures of center.

**Steps**

1. Filter `food_consumption` to get the rows where `food_category` is `"rice"`.
2. Create a histogram using `ggplot2` of `co2_emission` for rice.

```{r}
# Load packages
library(ggplot2)

food_consumption  |> 
  # 1. Filter for rice food category
  filter(food_category == "rice")  |> 
  # 2. Create histogram of co2_emission
  ggplot(aes(co2_emission)) +
    geom_histogram()
```

> *Question*
> ---
> Take a look at the histogram of the CO2 emissions for rice you just calculated. Which of the following terms best describes the shape of the data?<br>
> <br>
> ⬜ No skew<br>
> ⬜ Left-skewed<br>
> ✅ Right-skewed<br>

3. Filter `food_consumption` to get the rows where `food_category` is `"rice"`.

```{r}
food_consumption %>%
  # Filter for rice food category
  filter(food_category == "rice")  |> 
  # Create histogram of co2_emission
  ggplot(aes(co2_emission)) +
    geom_histogram()
```

4. Summarize the data to get the mean and median of `co2_emission`, calling them `mean_co2` and `median_co2`.

```{r}
food_consumption  |> 
  # Filter for rice food category
  filter(food_category == "rice") |> 
  # Get mean_co2 and median_co2
  summarise(mean_co2   = mean(co2_emission),
            median_co2 = median(co2_emission))
```

> *Question*
> ---
> Given the skew of this data, what measure of central tendency best summarizes the kilograms of CO2 emissions per person per year for rice?<br>
> <br>
> ⬜ Mean<br>
> ✅ Median<br>
> ⬜ Both mean and median<br>

Great work! The mean is substantially higher than the median since it's being pulled up by the high values over 100 kg/person/year.

## Measures of spread

Theory. Coming soon ...


**1. Measures of spread**

In this lesson, we'll talk about another set of summary statistics: measures of spread.

**2. What is spread?**

Spread is just what it sounds like - it describes how spread apart or close together the data points are. Just like measures of center, there are a few different measures of spread.

**3. Variance**

The first measure, variance, measures the average distance from each data point to the data's mean.

**4. Variance**

To calculate the variance, we start by calculating the distance between each point and the mean, so we get one number for every data point.

**5. Variance**

We then square each distance and then add them all together.

**6. Variance**

Finally, we divide the sum of squared distances by the number of data points minus 1, giving us the variance. The higher the variance, the more spread out the data is. It's important to note that the units of variance are squared, so in this case, it's 19.8 hours squared.We can calculate the variance in one step using the var function.

**7. Standard deviation**

The standard deviation is another measure of spread, calculated by taking the square root of the variance. It can also be calculated using the sd function. The nice thing about standard deviation is that the units are usually easier to understand since they're not squared. It's easier to wrap your head around 4 and a half hours than 19.8 hours squared.

**8. Mean absolute deviation**

Mean absolute deviation takes the absolute value of the distances to the mean, and then takes the mean of those differences. While this is similar to standard deviation, it's not exactly the same. Standard deviation squares distances, so longer distances are penalized more than shorter ones, while mean absolute deviation penalizes each distance equally. One isn't better than the other, but SD is more common than MAD.

**9. Quartiles**

Before we discuss the next measure of spread, let's quickly talk about quartiles. Quartiles split up the data into four equal parts. Here, we call the quantile function to get the quartiles of the data. This means that 25% of the data is between 1-point-9 and 7-point-85, another 25% is between 7-point-85 and 10-point-10, and so on. This means that the second quartile splits the data in two, with 50% of the data below it and 50% of the data above it, so it's exactly the same as the median.

**10. Boxplots use quartiles**

The boxes in box plots represent quartiles. The bottom of the box is the first quartile, and the top of the box is the third quartile. The middle line is the second quartile, or the median.

**11. Quantiles**

Quantiles, also called percentiles, are a generalized version of quartile, so they can split data into 5 pieces or ten pieces, for example. By default, the quantile function returns the quartiles of the data, but we can adjust this using the probs argument, which takes in a vector of proportions.Here, we split the data in five equal pieces. We can also use the seq function as a shortcut, which takes in the lowest number, the highest number, and the number we want to jump by. We can compute the same quantiles using seq from zero to one, jumping by 0-point-2.

**12. Interquartile range (IQR)**

The interquartile range, or IQR, is another measure of spread. It's the distance between the 25th and 75th percentile, which is also the height of the box in a boxplot. We can calculate it using the quantile function to get 5-point-9 hours.

**13. Outliers**

Outliers are data points that are substantially different from the others. But how do we know what a substantial difference is?A rule that's often used is that any data point less than the first quartile minus 1.5 times the IQR is an outlier, as well as any point greater than the third quartile plus 1.5 times the IQR.

**14. Finding outliers**

To find outliers, we'll start by calculating the IQR of the mammals' body weights. We can then calculate the lower and upper thresholds following the formulas from the previous slide.We can now filter the data frame to find mammals whose body weight is above or below the thresholds. We can see that there are eleven body weight outliers in this dataset, including the cow and the Asian elephant.

**15. Let's practice!**

Time to practice measuring spread and finding outliers.

## Quartiles, quantiles, and quintiles

Quantiles are a great way of summarizing numerical data since they can be used to measure center and spread, as well as to get a sense of where a data point stands in relation to the rest of the dataset. For example, you might want to give a discount to the 10% most active users on a website.

In this exercise, you'll calculate quartiles, quintiles, and deciles, which split up a dataset into 4, 5, and 10 pieces, respectively. 

**Steps**

1. Calculate the quartiles of the `co2_emission` column of `food_consumption`.

```{r}
# Calculate the quartiles of co2_emission
quantile(food_consumption$co2_emission)
```

2. Calculate the six quantiles that split up the data into 5 pieces (quintiles) of the `co2_emission` column of `food_consumption`.

```{r}
# Calculate the quintiles of co2_emission
quantile(food_consumption$co2_emission, probs = seq(0,1,0.2))
```

3. Calculate the eleven quantiles of `co2_emission` that split up the data into ten pieces (deciles).

```{r}
# Calculate the deciles of co2_emission
quantile(food_consumption$co2_emission, probs = seq(0,1,0.1))
```

Those are some high-quality quantiles! While calculating more quantiles gives you a more detailed look at the data, it also produces more numbers, making the summary more difficult to quickly understand.

## Variance and standard deviation

Variance and standard deviation are two of the most common ways to measure the spread of a variable, and you'll practice calculating these in this exercise. Spread is important since it can help inform expectations. For example, if a salesperson sells a mean of 20 products a day, but has a standard deviation of 10 products, there will probably be days where they sell 40 products, but also days where they only sell one or two. Information like this is important, especially when making predictions.

Both `dplyr` and `ggplot2` are loaded, and `food_consumption` is available.

**Steps**

1. Calculate the variance and standard deviation of `co2_emission` for each `food_category` by grouping by and summarizing variance as `var_co2` and standard deviation as `sd_co2`.

```{r}
# 1. Calculate variance and sd of co2_emission for each food_category
food_consumption %>% 
  group_by(food_category) %>% 
  summarise(var_co2 = var(co2_emission),
     sd_co2 = sd(co2_emission))
```

2. Create a histogram of `co2_emission` for each `food_category` using `facet_wrap()`.

```{r}
# 2. Plot food_consumption with co2_emission on x-axis
ggplot(food_consumption, aes(co2_emission)) +
  # Create a histogram
  geom_histogram() +
  # Create a separate sub-graph for each food_category
  facet_wrap(~ food_category)
```

Superb spread measurement! Beef has the biggest amount of variation in its CO$_2$ emissions, while eggs, nuts, and soybeans have relatively small amounts of variation.

## Finding outliers using IQR

Outliers can have big effects on statistics like mean, as well as statistics that rely on the mean, such as variance and standard deviation. Interquartile range, or IQR, is another way of measuring spread that's less influenced by outliers. IQR is also often used to find outliers. If a value is less than Q1 - 1.5 * IQR or greater than Q3 + 1.5 * IQR, it's considered an outlier. In fact, this is how the lengths of the whiskers in a `ggplot2` box plot are calculated.

<img src="https://assets.datacamp.com/production/repositories/5758/datasets/ca7e6e1832be7ec1842f62891815a9b0488efa83/Screen%20Shot%202020-04-28%20at%2010.04.54%20AM.png" alt="Diagram of a box plot showing median, quartiles, and outliers">

In this exercise, you'll calculate IQR and use it to find some outliers.

**Steps**

1. Calculate the total `co2_emission` per country by grouping by country and taking the sum of `co2_emission`. Call the sum `total_emission` and store the resulting data frame as `emissions_by_country`.

```{r}
# Calculate total co2_emission per country: emissions_by_country
emissions_by_country <- food_consumption %>%
  group_by(country) %>%
  summarise(total_emission = sum(co2_emission))

emissions_by_country
```

2. Compute the first and third quartiles of `total_emission` and store these as `q1` and `q3`.
3. Calculate the interquartile range of `total_emission` and store it as `iqr`.
4. Calculate the lower and upper cutoffs for outliers of `total_emission`, and store these as `lower` and `upper`.

```{r}
# 2. / 3. Compute the first and third quantiles and IQR of total_emission
q1  <- quantile(emissions_by_country$total_emission, probs = 0.25)
q3  <- quantile(emissions_by_country$total_emission, probs = 0.75)
iqr <- q3 - q1

# Calculate the lower and upper cutoffs for outliers
lower <- q1 - 1.5 * iqr
upper <- q3 + 1.5 * iqr
```

5. Use `filter()` to get countries with a `total_emission` greater than the `upper` cutoff **or** a `total_emission` less than the `lower` cutoff.

```{r}
# 5. Filter emissions_by_country to find outliers
emissions_by_country %>%
  filter(total_emission > upper | total_emission < lower)
```

Outstanding outlier detection! It looks like Argentina has a substantially higher amount of CO$_2$ emissions per person than other countries in the world.

# 2. Random Numbers and Probability

In this chapter, you'll learn how to generate random samples and measure chance using probability. You'll work with real-world sales data to calculate the probability of a salesperson being successful. Finally, you’ll use the binomial distribution to model events with binary outcomes.

## What are the chances?

Theory. Coming soon ...


**1. What are the chances?**

People talk about chance pretty frequently, like what are the chances of closing a sale, of rain tomorrow, or of winning a game? But how exactly do we measure chance?

**2. Measuring chance**

We can measure the chances of an event using probability. We can calculate the probability of some event by taking the number of ways the event can happen and dividing it by the total number of possible outcomes.For example, if we flip a coin, it can land on either heads or tails. To get the probability of the coin landing on heads, we divide the 1 way to get heads by the two possible outcomes, heads and tails. This gives us one half, or a fifty percent chance of getting heads.Probability is always between zero and 100 percent. If the probability of something is zero, it's impossible, and if the probability of something is 100%, it will certainly happen.

**3. Assigning salespeople**

Let's look at a more complex scenario. There's a meeting coming up with a potential client, and we want to send someone from the sales team to the meeting. We'll put each person's name on a ticket in a box and pull one out randomly to decide who goes to the meeting.

**4. Assigning salespeople**

Brian's name gets pulled out. The probability of Brian being selected is one out of four, or 25%.

**5. Sampling from a data frame**

We can recreate this scenario in R using dplyr's sample_n function, which takes in a data frame and the number of rows we want to pull out, which is only 1 in this case.However, if we run the same thing again, we may get a different row since sample_n chooses randomly. If we want to show the team how we picked Brian, this won't work well.

**6. Setting a random seed**

To ensure we get the same results when we run the script in front of the team, we'll set the random seed using set-dot-seed. The seed is a number that R's random number generator uses as a starting point, so if we orient it with a seed number, it will generate the same random value each time. The number itself doesn't matter. We could use 5, 139, or 3 million. The only thing that matters is that we use the same seed the next time we run the script. Now, we, or one of the sales-team members, can run this code over and over and get Brian every time.

**7. A second meeting**

Now there's another potential client who wants to meet at the same time, so we need to pick another salesperson. Brian already has been picked and he can't be in two meetings at once, so we'll pick between the remaining three. This is called sampling without replacement, since we aren't replacing the name we already pulled out.

**8. A second meeting**

This time, Claire is picked, and the probability of this is one out of three, or about 33%.

**9. Sampling twice in R**

To recreate this in R, we can pass 2 into sample_n, which will give us 2 rows.

**10. Sampling with replacement**

Now let's say the two meetings are happening on different days, so the same person could attend both. In this scenario, we need to return Brian's name to the box after picking it. This is called sampling with replacement.

**11. Sampling with replacement**

Claire gets picked for the second meeting, but this time, the probability of picking her is 25%.

**12. Sampling with replacement in R**

To sample with replacement, set the replace argument of sample_n to TRUE.If there were 5 meetings, all at different times, it's possible to pick some rows multiple times since we're replacing them each time.

**13. Independent events**

Let's quickly talk about independence. Two events are independent if the probability of the second event isn't affected by the outcome of the first event. For example, if we're sampling with replacement, the probability

**14. Independent events**

that Claire is picked second is 25%, no matter who gets picked first.In general, when sampling with replacement, each pick is independent.

**15. Dependent events**

Similarly, events are considered dependent when the outcome of the first changes the probability of the second.If we sample without replacement, the probability that Claire is picked second depends on who gets picked first.

**16. Dependent events**

If Claire is picked first, there's 0% probability that Claire will be picked second.

**17. Dependent events**

If someone else is picked first, there's a 33% probability Claire will be picked second.In general, when sampling without replacement, each pick is dependent.

**18. Let's practice!**

Head over to the exercises!

## With or without replacement?

In the video, you learned about two different ways of taking samples: with replacement and without replacement. Although it isn't always easy to tell which best fits various situations, it's important to correctly identify this so that any probabilities you report are accurate. In this exercise, you'll put your new knowledge to the test and practice figuring this out.

For each scenario, decide whether it's sampling with replacement or sampling without replacement.

| With replacement | Without replacement |
| --- | --- |
| Flipping a coin 3 times | Randomly selecting 5 products from the assembly line to test for quality assurance |
| Rolling a die twice | From a deck of cards, dealing 3 players 7 cards each | 
| | Randomly picking 3 people to work on the weekend from a group of 20 people | 

## Calculating probabilities

You're in charge of the sales team, and it's time for performance reviews, starting with Amir. As part of the review, you want to randomly select a few of the deals that he's worked on over the past year so that you can look at them more deeply. Before you start selecting deals, you'll first figure out what the chances are of selecting certain deals.

Recall that the probability of an event can be calculated by

<img src="https://render.githubusercontent.com/render/math?math={\color{red}P(x)=\frac{%23ways event can happen}{total %23 of possible outcomes}}">

**Steps**

1. Count the number of deals Amir worked on for each `product` type.

```{r}
amir_deals <-  readRDS("data/amir_deals.rds")

# Count the deals for each product
amir_deals %>%
  count(product, sort = T)
```

2. Create a new column called prob by dividing n by the total number of deals Amir worked on.

```{r}
# Calculate probability of picking a deal with each product
amir_deals %>%
  count(product, sort = T) %>%
  mutate(prob = n/sum(n))
```

> *Question*
> ---
> If you randomly select one of Amir's deals, what's the probability that the deal will involve Product C?<br>
> <br>
> ⬜ 15%<br>
> ⬜ 80.43%<br>
> ✅ 8.43%<br>
> ⬜ 22.5%<br>

Perfect probability calculations! Now that you know what the chances are, it's time to start sampling.

## Sampling deals

In the previous exercise, you counted the deals Amir worked on. Now it's time to randomly pick five deals so that you can reach out to each customer and ask if they were satisfied with the service they received. You'll try doing this both with and without replacement.

Additionally, you want to make sure this is done randomly and that it can be reproduced in case you get asked how you chose the deals, so you'll need to set the random seed before sampling from the deals.

**Steps**

1. Set the random seed to `31`.
2. Take a sample of 5 deals **without** replacement.

```{r}
# Set random seed to 31
set.seed(31)

# Sample 5 deals without replacement
amir_deals %>%
  sample_n(5)
```

3. Take a sample of 5 deals **with** replacement.

```{r}
# Set random seed to 31
set.seed(31)

# Sample 5 deals with replacement
amir_deals %>%
  sample_n(5, replace = T)
```

> *Question*
> ---
> What type of sampling is better to use for this situation?<br>
> <br>
> ⬜ With replacement<br>
> ✅ Without replacement<br>
> ⬜ It doesn't matter<br>

Spectactular sampling! It's important to consider how you'll take a sample since there's no one-size-fits-all way to sample, and this can have an effect on your results.

## Discrete distributions

Theory. Coming soon ...


**1. Discrete distributions**

In this lesson, we'll take a deeper dive into probability and begin looking at probability distributions.

**2. Rolling the dice**

Let's consider rolling a standard, six-sided die.

**3. Rolling the dice**

There are six numbers, or six possible outcomes, and every number has one sixth, or about a 17 percent chance of being rolled. This is an example of a probability distribution.

**4. Choosing salespeople**

This is similar to the scenario from earlier, except we had names instead of numbers. Just like rolling a die, each outcome, or name, had an equal chance of being chosen.

**5. Probability distribution**

A probability distribution describes the probability of each possible outcome in a scenario.We can also talk about the expected value of a distribution, which is the mean of a distribution. We can calculate this by multiplying each value by its probability (one sixth in this case) and summing, so the expected value of rolling a fair die is 3.5.

**6. Visualizing a probability distribution**

We can visualize this using a barplot, where each bar represents an outcome, and each bar's height represents the probability of that outcome.

**7. Probability = area**

We can calculate probabilities of different outcomes by taking areas of the probability distribution.For example, what's the probability that our die roll is less than or equal to 2? To figure this out, we'll take the area of each bar representing an outcome of 2 or less.

**8. Probability = area**

Each bar has a width of 1 and a height of one sixth, so the area of each bar is one sixth. We'll sum the areas for 1 and 2, to get a total probability of one third.

**9. Uneven die**

Now let's say we have a die where the two got turned into a three. This means that we now have a 0% chance of getting a 2, and a 33% chance of getting a 3. To calculate the expected value of this die, we now multiply 2 by 0, since it's impossible to get a 2, and 3 by its new probability, one third. This gives us an expected value that's slightly higher than the fair die.

**10. Visualizing uneven probabilities**

When we visualize these new probabilities, the bars are no longer even.

**11. Adding areas**

With this die, what's the probability of getting something less than or equal to 2? There's a one sixth probability of getting 1, and zero probability of getting 2,

**12. Adding areas**

which sums to one sixth.

**13. Discrete probability distributions**

The probability distributions you've seen so far are both discrete probability distributions, since they represent situations with discrete outcomes. Recall from chapter 1 that discrete variables can be thought of as counted variables. In the case of a die, we're counting dots, so we can't roll a 1-point-5 or 4-point-3.When all outcomes have the same probability, like a fair die, this is a special distribution called a discrete uniform distribution.

**14. Sampling from discrete distributions**

Just like we sampled names from a box, we can do the same thing with probability distributions like the ones we've seen. Here's a data frame called die that represents a fair die, and its expected value is 3-point-5.We'll sample from it 10 times to simulate 10 rolls. Notice that we sample with replacement so that we're sampling from the same distribution every time.

**15. Visualizing a sample**

We can visualize the outcomes of the ten rolls using a histogram, setting the number of bins to 6 since there are 6 possible outcomes.

**16. Sample distribution vs. theoretical distribution**

Notice that we have different numbers of 1's, 2's, 3's, and so on since the sample was random, even though on each roll we had the same probability of rolling each number. The mean of our sample is 3-point-0, which isn't super close to the 3-point-5 we were expecting.

**17. A bigger sample**

If we roll the die 100 times, the distribution of the rolls looks a bit more even, and the mean is closer to 3-point-5.

**18. An even bigger sample**

If we roll 1000 times, it looks even more like the theoretical probability distribution and the mean closely matches 3-point-5.

**19. Law of large numbers**

This is called the law of large numbers, which is the idea that as the size of your sample increases, the sample mean will approach the theoretical mean.

**20. Let's practice!**

Time to solidify your knowledge of probability distributions.

## Creating a probability distribution

A new restaurant opened a few months ago, and the restaurant's management wants to optimize its seating space based on the size of the groups that come most often. On one night, there are 10 groups of people waiting to be seated at the restaurant, but instead of being called in the order they arrived, they will be called randomly. In this exercise, you'll investigate the probability of groups of different sizes getting picked first. Data on each of the ten groups is contained in the `restaurant_groups` data frame.

Remember that expected value can be calculated by multiplying each possible outcome with its corresponding probability and taking the sum.

**Steps**

1. Create a histogram of the `group_size` column of `restaurant_groups`, setting the number of bins to `5`.

```{r}
# restaurant groups
restaurant_groups <- tibble(
  group_id   = c(1:10),
  group_size = c(2, 4, 6, 2, 2, 2, 3, 2, 4, 2)
)

# 1. Create a histogram of group_size
restaurant_groups |> 
  ggplot(aes(group_size)) +
    geom_histogram(bins = 5)
```
2. Count the number of each `group_size` in `restaurant_groups`, then add a column called `probability` that contains the probability of randomly selecting a group of each size. Store this in a new data frame called `size_distribution`.

```{r}
# 2. Create probability distribution
size_distribution <- restaurant_groups %>%
  # Count number of each group size
  count(group_size) %>%
  # Calculate probability
  mutate(probability = n / sum(n))

size_distribution
```

3. Calculate the expected value of the `size_distribution`, which represents the expected group size.

```{r}
# 3. Calculate expected group size
expected_val <- sum(size_distribution$probability *
                    size_distribution$group_size)
expected_val
```

4. Calculate the probability of randomly picking a group of 4 or more people by filtering and summarizing.

```{r}
# 4. Calculate probability of picking group of 4 or more
size_distribution %>%
  # Filter for groups of 4 or larger
  filter(group_size >= 4 ) %>%
  # Calculate prob_4_or_more by taking sum of probabilities
  summarise(prob_4_or_more = sum(probability))
```

Dexterous distribution utilization! You'll continue to build upon these skills since many statistical tests and methods use probability distributions as their foundation.

## Identifying distributions

::: {.callout-caution}
Unsolved issues after ggplot update:

! Problem while computing aesthetics.
ℹ Error occurred in the 1st layer.
Caused by error in `scales_add_defaults()`:
! could not find function "scales_add_defaults"
:::

```{r}
library(patchwork)
p1 <- readRDS("data/p1.rds"); p2 <- readRDS("data/p2.rds"); p3 <- readRDS("data/p3.rds")
# p1 + p2 + p3
```

> *Question*
> ---
> Which sample is most likely to have been taken from a uniform distribution?<br>
> <br>
> ✅ A<br>
> ⬜ B<br>
> ⬜ C<br>

Impressive identification! Since the histogram depicts a sample and not the actual probability distribution, each outcome won't happen the exact same number of times due to randomness, but they're similar in number.

## Expected value vs. sample mean

```{r, fig.height=2, out.width="100%"}
library(purrr)

sample_sizes <- c(10,100,1000,5000, 10000)

sample_sizes |> 
  
  # Set names, so that the data can be faceted according to the names
  set_names() |> 
  # map rdunif against the sample sizes
  map(rdunif, b = 9, a = 1) |> 
  map(as_tibble) |> 
  # Stack data
  bind_rows(.id = "sample_size") |> 
  # Change order
  mutate(sample_size = sample_size |> forcats::fct_inorder()) |> 
  # Plot
  ggplot(aes(value)) + 
        geom_histogram(binwidth = 1) + 
        facet_wrap(~sample_size, scales = "free", nrow = 1)
```

The code above will take a sample from a discrete uniform distribution, which includes the numbers 1 through 9, and calculate the sample's mean. You can adjust the size of the samples by changing the values of `sample_sizes`. Note that the expected value of this distribution is 5.

> *Question*
> ---
> A sample is taken, and you win twenty dollars if the sample's mean is less than 4. There's a catch: you get to pick the sample's size. Which sample size is most likely to win you the twenty dollars?<br>
> <br>
> ✅ 10<br>
> ⬜ 100<br>
> ⬜ 1000<br>
> ⬜ 5000<br>
> ⬜ 10000<br>

Nice work! Since the sample mean will likely be closer to 5 (the expected value) with larger sample sizes, you have a better chance of getting a sample mean further away from 5 with a smaller sample. 

## Continuous distributions

Theory. Coming soon ...


**1. Continuous distributions**

We can use discrete distributions to model situations that involve discrete or countable variables, but how can we model continuous variables?

**2. Waiting for the bus**

Let's start with an example. The city bus arrives every twelve minutes, so if you show up at a random time, you could wait anywhere from 0 minutes if you arrive just as the bus pulls in, up to 12 minutes if you arrive as the bus leaves.

**3. Continuous uniform distribution**

Let's model this scenario with a probability distribution. There are an infinite number of minutes we could wait since we could wait 1 minute, 1-point-5 minutes, 1-point-53 minutes, and so on, so we can't create individual blocks like we could with a discrete variable.

**4. Continuous uniform distribution**

Instead, we'll use a continuous line to represent probability. The line is flat since there's the same probability of waiting any time from 0 to 12 minutes. This is called the continuous uniform distribution

**5. Probability still = area**

Now that we have our distribution, let's figure out what the probability is that we'll wait between 4 and 7 minutes. Just like with discrete distributions, we can take the area from 4 to 7 to calculate probability.

**6. Probability still = area**

The width of this rectangle is 7 minus 4 which is 3. The height is one twelfth.

**7. Probability still = area**

Multiplying those together to get area, we get 3/12 or 25%.

**8. Uniform distribution in R**

Let's use the uniform distribution in R to calculate the probability of waiting 7 minutes or less. We'll pass 7 into punif. It also takes in a min and a max, which in our case is 0 and 12. The probability of waiting less than 7 minutes is about 58%.

**9. lower.tail**

If we want the probability of waiting more than 7 minutes, set the lower-dot-tail argument to FALSE.

**10. Combining multiple punif() calls**

But how do we calculate the probability of waiting 4 to 7 minutes using R?

**11. Combining multiple punif() calls**

We can start with the probability of waiting less than 7 minutes,

**12. Combining multiple punif() calls**

then subtract the probability of waiting less than 4 minutes. This gives us 25%.

**13. Total area = 1**

To calculate the probability of waiting between 0 and 12 minutes, we multiply 12 by 1/12, which is 1.

**14. Total area = 1**

or 100%. This makes sense since we're certain we'll wait anywhere from 0 to 12 minutes.

**15. Other continuous distributions**

Continuous distributions can take forms other than uniform where some values have a higher probability than others.

**16. Other continuous distributions**

No matter the shape of the distribution, the area beneath it must always equal 1.

**17. Other special types of distributions**

This will also be true of other distributions you'll learn about later on in the course, like the normal distribution or Poisson distribution, which can be used to model many real-life situations.

**18. Let's practice!**

Time to practice working with continuous distributions.



## Which distribution?

At this point, you've learned about the two different variants of the uniform distribution: the discrete uniform distribution, and the continuous uniform distribution. In this exercise, you'll decide which situations follow which distribution.

<img src="https://assets.datacamp.com/production/repositories/5758/datasets/fe928d4c840f66544c6228b8b755e9bf15361b9f/Screen%20Shot%202020-05-04%20at%205.18.16%20PM.png" alt="Illustration of discrete and continuous uniform distributions">

**Steps**

Map each situation to the probability distribution it would best be modeled by.

| Discrete Uniform | Continuous Unifrom | Other |
| --- | --- | --- |
| The ticket number of a raffle winner, assuming there is one ticket for each number from 1 to 100. | The time you'll have to wait for a geyser to erupt if you show up at a random time, knowing that the geyser erupts exactly every ten minutes. | The height of a random person. |
| The outcome of rolling a 4-sided die. | The time of day a baby will be born. | |

## Data back-ups

The sales software used at your company is set to automatically back itself up, but no one knows exactly what time the back-ups happen. It is known, however, that back-ups happen exactly every 30 minutes. Amir comes back from sales meetings at random times to update the data on the client he just met with. He wants to know how long he'll have to wait for his newly-entered data to get backed up. Use your new knowledge of continuous uniform distributions to model this situation and answer Amir's questions.

**Steps**

1. To model how long Amir will wait for a back-up using a continuous uniform distribution, save his lowest possible wait time as `min` and his longest possible wait time as `max`. Remember that back-ups happen every 30 minutes.
2. Calculate the probability that Amir has to wait less than 5 minutes, and store in a variable called `prob_less_than_5`.

```{r}
# 1. Min and max wait times for back-up that happens every 30 min
min <- 0
max <- 30

# 2. Calculate probability of waiting less than 5 mins
prob_less_than_5 <- punif(5, min = min, max = max)
prob_less_than_5
```

3. Calculate the probability that Amir has to wait more than 5 minutes, and store in a variable called `prob_greater_than_5`.

```{r}
# 3. Calculate probability of waiting more than 5 mins
prob_greater_than_5 <- punif(5, min = min, max = max, lower.tail = F)
prob_greater_than_5
```

4. Calculate the probability that Amir has to wait between 10 and 20 minutes, and store in a variable called `prob_between_10_and_20`.

```{r}
# 4. Calculate probability of waiting 10-20 mins
prob_between_10_and_20 <- punif(20, min = min, max = max) - punif(10, min = min, max = max)
prob_between_10_and_20
```

Wonderful wait-time calculations! There's a 33% chance that Amir will wait 10-20 minutes. In the next exercise, you'll make sure this calculation holds up by simulating some wait times.

## Simulating wait times

To give Amir a better idea of how long he'll have to wait, you'll simulate Amir waiting 1000 times and create a histogram to show him what he should expect. Recall from the last exercise that his minimum wait time is 0 minutes and his maximum wait time is 30 minutes.

**Steps**

1. Set the random seed to `334`.
2. Generate 1000 wait times from the continuous uniform distribution that models Amir's wait time. Add this as a new column called `time` in the `wait_times` data frame.

```{r}
wait_times <- tibble(simulation_nb=c(1:1000))

# 1. Set random seed to 334
set.seed(334)

# 2. Generate 1000 wait times between 0 and 30 mins, save in time column
wait_times %>%
  mutate(time = runif(1000, min = 0, max = 30))
```

* 3. Create a histogram of the simulated wait times with 30 bins.

```{r}
# 1. Set random seed to 334
set.seed(334)

# 3. Generate 1000 wait times between 0 and 30 mins, save in time column
wait_times %>%
  mutate(time = runif(1000, min = 0, max = 30)) %>%
  # Create a histogram of simulated times
  ggplot(aes(time)) +
  geom_histogram()
```

Superb simulating! Unless Amir figures out exactly what time each backup happens, he won't be able to time his data entry so it gets backed up sooner, but it looks like he'll wait about 15 minutes on average.

## The binomial distribution

Theory. Coming soon ...


**1. The binomial distribution**

It's time to further expand your toolbox of distributions. In this video, you'll learn about the binomial distribution.

**2. Coin flipping**

We'll start by flipping a coin, which has two possible outcomes, heads or tails, each with a probability of 50%.

**3. Binary outcomes**

This is just one example of a binary outcome, or an outcome with two possible values. We could also represent these outcomes as a 1 and a 0, a success or a failure, and a win or a loss.

**4. A single flip**

In R, we can simulate this using the rbinom function, which takes in the number of trials, or times we want to flip, the number of coins we want to flip, and the probability of heads or success. This will return a 1, which we'll count as a head, or a 0, which we'll count as tails.We can use rbinom 1, 1, 0-point-5 to flip 1 coin 1 time with a 50% probability of heads.

**5. One flip many times**

To perform eight coin flips, we can change the first argument to an 8, which will give us eight flips of 1 coin with a 50% chance of heads.This gives us a set of 8 ones and zeros.

**6. Many flips one time**

If we swap the first two arguments, we simulate one flip of eight coins. This gives us one number, which is the total number of heads or successes.

**7. Many flips many times**

Similarly, we can pass 10 and 3 into rbinom to simulate 10 flips of 3 coins. This returns 10 numbers, each representing the total number of heads from each set of flips.

**8. Other probabilities**

We could also have a coin that's heavier on one side than the other, so the probability of getting heads is only 25%. To simulate flips with this coin, we'll adjust the third argument of rbinom to 0-point-25. The result has lower numbers, since getting multiple heads isn't as likely with the new coin.

**9. Binomial distribution**

The binomial distribution describes the probability of the number of successes in a sequence of independent trials. In other words, it can tell us the probability of getting some number of heads in a sequence of coin flips. Note that this is a discrete distribution since we're working with a countable outcome.The binomial distribution can be described using two parameters, n and p. n represents the total number of trials being performed. n and p are also the second and third arguments of rbinom.Here's what the distribution looks like for 10 coins. We have the biggest chance of getting 5 heads total, and a much smaller chance of getting 0 heads or 10 heads.

**10. What's the probability of 7 heads?**

To get the probability of getting 7 heads out of 10 coins, we can use dbinom. The first argument is the number of heads or successes. The second argument is the number of trials, n, and the third is the probability of success, p.If we flip 10 coins, there's about a 12% chance that 7 of them will be heads.

**11. What's the probability of 7 or fewer heads?**

pbinom gives the probability of getting a number of successes less than or equal to the first argument. The probability of getting 7 or fewer heads out of 10 coins is about 95%.

**12. What's the probability of more than 7 heads?**

We can use the lower-dot-tail argument to get the probability of a number of successes greater than the first argument. Note that this is the same as 1 minus the same pbinom call from the previous slide.

**13. Expected value**

The expected value of the binomial distribution can be calculated by multiplying n times p. The expected number of heads we'll get from flipping 10 coins is 10 times 0-point-5, which is 5.

**14. Independence**

It's important to remember that in order for the binomial distribution to apply, each trial must be independent, so the outcome of one trial shouldn't have an effect on the next.For example, if we're picking randomly from these cards with zeros and ones, we have a 50-50 chance of getting a 0 or a 1.

**15. Independence**

But since we're sampling without replacement, the probabilities for the second trial are different due to the outcome of the first trial. Since these trials aren't independent, we can't calculate accurate probabilities for this situation using the binomial distribution.

**16. Let's practice!**

Time to explore binary outcomes using the binomial distribution.

## Simulating sales deals

Assume that Amir usually works on 3 deals per week, and overall, he wins 30% of deals he works on. Each deal has a binary outcome: it's either lost, or won, so you can model his sales deals with a binomial distribution. In this exercise, you'll help Amir simulate a year's worth of his deals so he can better understand his performance.

**Steps**

1. Set the random seed to 10 and simulate a single deal.

```{r}
# 1. Set random seed to 10
set.seed(10)

# 1. Simulate a single deal
rbinom(1, 1, 0.3)
```

2. Simulate a typical week of Amir's deals, or one week of 3 deals.

```{r}
# 1. Set random seed to 10
set.seed(10)

# 2. Simulate a single deal
rbinom(1, 3, 0.3)
```

3. Simulate a year's worth of Amir's deals, or 52 weeks of 3 deals each, and store in `deals`.
4. Calculate the mean number of deals he won per week.

```{r}
# Set random seed to 10
set.seed(10)

# Simulate 52 weeks of 3 deals
deals <- rbinom(52,3,0.3)

# Calculate mean deals won per week
mean(deals)
```

Brilliant binomial simulation! In this simulated year, Amir won 0.8 deals on average each week.

## Calculating binomial probabilities

Just as in the last exercise, assume that Amir wins 30% of deals. He wants to get an idea of how likely he is to close a certain number of deals each week. In this exercise, you'll calculate what the chances are of him closing different numbers of deals using the binomial distribution.


**Steps**

1. What's the probability that Amir closes all 3 deals in a week?

```{r}
# Probability of closing 3 out of 3 deals
dbinom(3,3,0.3)
```

2. What's the probability that Amir closes 1 or fewer deals in a week?

```{r}
# Probability of closing <= 1 deal out of 3 deals
pbinom(1,3,0.3)
```

3. What's the probability that Amir closes more than 1 deal?

```{r}
# Probability of closing > 1 deal out of 3 deals
pbinom(1,3,0.3,lower.tail=F)
```


Powerful probability calculations! Amir has about a 22% chance of closing more than one deal in a week.

## How many sales will be won?

Now Amir wants to know how many deals he can expect to close each week if his win rate changes. Luckily, you can use your binomial distribution knowledge to help him calculate the expected value in different situations. Recall from the video that the expected value of a binomial distribution can be calculated by (n * p).

**Steps**

1. Calculate the expected number of sales out of the **3** he works on that Amir will win each week if he maintains his 30% win rate.

```{r}
# Expected number won with 30% win rate
won_30pct <- 3 * 0.30
won_30pct
```

2. Calculate the expected number of sales out of the 3 he works on that he'll win if his win rate drops to 25%.

```{r}
# Expected number won with 25% win rate
won_25pct <- 3 * 0.25
won_25pct
```

3. Calculate the expected number of sales out of the 3 he works on that he'll win if his win rate rises to 35%.

```{r}
# Expected number won with 35% win rate
won_35pct <- 3 * 0.35
won_35pct
```

Excellent expectation experimentation! If Amir's win rate goes up by 5%, he can expect to close more than 1 deal on average each week.

# 3. More Distributions and the Central Limit Theorem

It’s time to explore one of the most important probability distributions in statistics, normal distribution. You’ll create histograms to plot normal distributions and gain an understanding of the central limit theorem, before expanding your knowledge of statistical functions by adding the Poisson, exponential, and t-distributions to your repertoire.

## The normal distribution

Theory. Coming soon ...


**1. The normal distribution**

The next probability distribution we'll discuss is the normal distribution. It's one of the most important probability distributions you'll learn about since a countless number of statistical methods rely on it, and it applies to more real-world situations than the distributions we've covered so far.

**2. What is the normal distribution?**

The normal distribution looks like this. Its shape is commonly referred to as a "bell curve". The normal distribution has a few important properties.

**3. Symmetrical**

First, it's symmetrical, so the left side is a mirror image of the right.

**4. Area = 1**

Second, just like any continuous distribution, the area beneath the curve is 1.

**5. Curve never hits 0**

Third, the probability never hits 0, even if it looks like it at the tail ends. Only 0-point-006% of its area is contained beyond the edges of this graph.

**6. Described by mean and standard deviation**

The normal distribution is described by its mean and standard deviation. Here is a normal distribution with a mean of 20 and standard deviation of 3, and here is a normal distribution with a mean of 0 and a standard deviation of 1. When a normal distribution has mean 0 and a standard deviation of 1, it's a special distribution called the standard normal distributionNotice how both distributions have the same shape,

**7. Described by mean and standard deviation**

but their axes have different scales.

**8. Areas under the normal distribution**

For the normal distribution, 68% of the area is within 1 standard deviation of the mean.

**9. Areas under the normal distribution**

95% of the area falls within 2 standard deviations of the mean,

**10. Areas under the normal distribution**

and 99-point-7% of the area falls within three standard deviations. This is sometimes called the 68-95-99-point-7 rule.

**11. Lots of histograms look normal**

There's lots of real-world data shaped like the normal distribution. For example, here is a histogram of the heights of women that participated in the National Health and Nutrition Examination Survey. The mean height is around 161 centimeters and the standard deviation is about 7 centimeters.

**12. Approximating data with the normal distribution**

Since this height data closely resembles the normal distribution, we can take the area under a normal distribution with mean 161 and standard deviation 7 to approximate what percent of women fall into different height ranges.

**13. What percent of women are shorter than 154 cm?**

For example, what percent of women are shorter than 154 centimeters? We can answer this using the pnorm function, which takes the area of the normal distribution less than some number. We pass in the number of interest, 154, as well as the mean and standard deviation of the normal distribution we're using. This gives us about 16% of women are shorter than 154 centimeters.

**14. What percent of women are taller than 154 cm?**

To find the percent of women taller than 154 centimeters, we can add lower-dot-tail equals FALSE, which will take the area to the right of the first argument.

**15. What percent of women are 154-157 cm?**

To get the percent of women between 154 and 157 centimeters tall we can take the area below 157 and subtract the area below 154, which leaves us the area between 154 and 157.

**16. What percent of women are 154-157 cm?**

which leaves us the area between 154 and 157.

**17. What height are 90% of women shorter than?**

We can also calculate percentages from heights using qnorm. To figure out what height 90% of women are shorter than, we pass 0-point-9 into qnorm along with the same mean and standard deviation we've been working with. This tells us that 90% of women are shorter than 170 centimeters tall.

**18. What height are 90% of women taller than?**

Similarly, we can figure out the height 90% of women are taller than by setting the lower-dot-tail argument of qnorm to FALSE.

**19. Generating random numbers**

Just like with other distributions, we can generate random numbers from a normal distribution using rnorm, passing in the sample size we want along with the distribution's mean and standard deviation. Here, we've generated 10 more random heights.

**20. Let's practice!**

Time to practice using the normal distribution!

## Distribution of Amir's sales

Since each deal Amir worked on (both won and lost) was different, each was worth a different amount of money. These values are stored in the `amount` column of `amir_deals` As part of Amir's performance review, you want to be able to estimate the probability of him selling different amounts, but before you can do this, you'll need to determine what kind of distribution the `amount` variable follows.

**Steps**

1. Create a histogram with 10 bins to visualize the distribution of the `amount`.

```{r}
# Histogram of amount with 10 bins
ggplot(amir_deals, aes(amount)) +
    geom_histogram(bins = 10)
```

> *Question*
> ---
> Which probability distribution do the sales amounts most closely follow?<br>
> <br>
> ⬜ Uniform<br>
> ⬜ Binomial<br>
> ✅ Normal<br>
> ⬜ None of the above<br>

Fabulous work! Now that you've visualized the data, you know that you can approximate probabilities of different `amount`s using the normal distribution.

## Probabilities from the normal distribution

Since each deal Amir worked on (both won and lost) was different, each was worth a different amount of money. These values are stored in the `amount` column of `amir_deals` and follow a normal distribution with a mean of 5000 dollars and a standard deviation of 2000 dollars. As part of his performance metrics, you want to calculate the probability of Amir closing a deal worth various amounts.

**Steps**

1. What's the probability of Amir closing a deal worth less than $7500?

```{r}
# Probability of deal < 7500
pnorm(7500, 5000, 2000)
```

2. What's the probability of Amir closing a deal worth more than $1000?

```{r}
# Probability of deal > 1000
pnorm(1000, 5000, 2000, lower.tail=F)
```

3. What's the probability of Amir closing a deal worth between $3000 and $7000?

```{r}
# Probability of deal between 3000 and 7000
pnorm(7000, 5000, 2000) - pnorm(3000, 5000, 2000)
```

4. What amount will 75% of Amir's sales be *more than*?

```{r}
# Calculate amount that 75% of deals will be more than
qnorm(0.75, 5000, 2000, lower.tail =F)
```

Nifty normal distribution usage! You know that you can count on Amir 75% of the time to make a sale worth at least $3651.02, and this information will be useful in making company-wide sales projections.

## Simulating sales under new market conditions

The company's financial analyst is predicting that next quarter, the worth of each sale will increase by 20% and the volatility, or standard deviation, of each sale's worth will increase by 30%. To see what Amir's sales might look like next quarter under these new market conditions, you'll simulate new sales amounts using the normal distribution and store these in the `new_sales` data frame, which has already been created for you.

**Steps**

1. Currently, Amir's average sale amount is $5000. Calculate what his new average amount will be if it increases by 20% and store this in `new_mean`.
2. Amir's current standard deviation is $2000. Calculate what his new standard deviation will be if it increases by 30% and store this in `new_sd`.
3. Add a new column called `amount` to the data frame `new_sales`, which contains 36 simulated amounts from a normal distribution with a mean of `new_mean` and a standard deviation of `new_sd`.

```{r}
new_sales <- tibble(sale_num = c(1:36))

# Calculate new average amount
new_mean <- 5000 * 1.2

# Calculate new standard deviation
new_sd <- 2000 * 1.3

# Simulate 36 sales
new_sales <- new_sales %>% 
  mutate(amount = rnorm(36, new_mean, new_sd))
```

4. Plot the distribution of the `new_sales` `amount`s using a histogram with 10 bins.

```{r}
# 4. Create histogram with 10 bins
ggplot(new_sales, aes(amount)) +
  geom_histogram(bins = 10)
```

Successful simulating! Although the average sale amount went up, the variation also increased, so it's not straightforward to decide whether these sales are better than his current ones. In the next exercise, you'll explore the effects of higher variation.

## Which market is better?

The key metric that the company uses to evaluate salespeople is the percent of sales they make over $1000 since the time put into each sale is usually worth a bit more than that, so the higher this metric, the better the salesperson is performing. 

Recall that Amir's current sales amounts have a mean of $5000 and a standard deviation of $2000, and Amir's predicted amounts in next quarter's market have a mean of $6000 and a standard deviation of $2600.

> *Question*
> ---
> Based only on the metric of **percent of sales over $1000**, does Amir perform better in the current market or the predicted market?<br>
> <br>
> ⬜ Amir performs much better in the current market.<br>
> ⬜ Amir performs much better in next quarter's predicted market.<br>
> ✅ Amir performs about equally in both markets.<br>

Great work! In the current market, Amir makes sales over $1000 about 97.7% of the time, and about 97.3% of the time in the predicted market, so there's not much of a difference. However, his average sale amount is higher in the predicted market, so the company may want to consider other metrics as well. 

## The central limit theorem

Theory. Coming soon ...


**1. The central limit theorem**

Now that you're familiar with the normal distribution, it's time to learn about what makes it so important.

**2. Rolling the dice 5 times**

Let's go back to our dice rolling example. We have a vector of the numbers 1 to 6 called die. To simulate rolling the die 5 times, we'll use the sample() function. sample() works the same way as sample_n(), except it samples from a vector instead of a data frame. We pass in the vector we want to sample from, the size of the sample, and set replace to TRUE.This gives us the results of 5 rolls.Now, we'll take the mean of the 5 rolls, which gives us 2.

**3. Rolling the dice 5 times**

If we roll another 5 times and take the mean, we get a different mean. If we do it again, we get another mean.

**4. Rolling the dice 5 times 10 times**

Let's repeat this 10 times: we'll roll 5 times and take the mean.To do this, we'll use replicate. We pass in 10 so that the process is repeated 10 times, followed by the snippet of code we want to be run, which is the rolling and averaging.This returns a vector of 10 different sample means.Let's plot these sample means.

**5. Sampling distributions**

A distribution of a summary statistic like this is called a sampling distribution. This distribution, specifically, is a sampling distribution of the sample mean.

**6. 100 sample means**

Now let's do this 100 times. If we look at the new sampling distribution, its shape somewhat resembles the normal distribution, even though the distribution of the die is uniform.

**7. 1000 sample means**

Let's take 1000 means. This sampling distribution more closely resembles the normal distribution.

**8. Central limit theorem**

This phenomenon is known as the central limit theorem, which states that a sampling distribution will approach a normal distribution as the number of trials increases.In our example, the sampling distribution became closer to the normal distribution as we took more and more sample means.It's important to note that the central limit theorem only applies when samples are taken randomly and are independent, for example, randomly picking sales deals with replacement.

**9. Standard deviation and the CLT**

The central limit theorem, or CLT, applies to other summary statistics as well.If we take the standard deviation of 5 rolls 1000 times, the sample standard deviations are distributed normally, centered around 1-point-9, which is the distribution's standard deviation.

**10. Proportions and the CLT**

Another statistic that the CLT applies to is proportion.Let's sample from the sales team 10 times with replacement and see how many draws have Claire as the outcome. In this case, 10% of draws were Claire. If we draw again, there are 40% Claires.

**11. Sampling distribution of proportion**

If we repeat this 1000 times and plot the distribution of the sample proportions, it resembles a normal distribution centered around 0-point-25, since Claire's name was on 25% of the tickets.

**12. Mean of sampling distribution**

Since these sampling distributions are normal, we can take their mean to get an estimate of a distribution's mean, standard deviation, or proportion.If we take the mean of our sample means from earlier, we get 3-point-48. That's pretty close to the expected value, which is 3-point-5!Similarly, the mean of the sample proportions of Claires isn't far off from 0-point-25.In these examples, we know what the underlying distributions look like, but if we don't, this can be a useful method for estimating characteristics of an underlying distribution.The central limit theorem also comes in handy when you have a huge population and don't have the time or resources to collect data on everyone. Instead, you can collect several smaller samples and create a sampling distribution to estimate what the mean or standard deviation is.

**13. Let's practice!**

Now, it's time to practice utilizing the central limit theorem.

## Visualizing sampling distributions

```{r}
# Function to resample distributions
resample_dist <- function(x, sample_mean = 100000, sample_size = 20) {
  replicate(n    = sample_mean,
            expr = mean(sample(x       = x,
                               size    = sample_size,
                               replace = TRUE)))
}

# Set seed
set.seed(1234)

n <- 4000
# 1 Exponential Distribution
x_exp    <- rexp(n = n, rate = 0.1)
dist_exp <- resample_dist(x_exp)

# 2 Uniform Distribution
x_unif    <- runif(n = n, min = 0, max = 1)
dist_unif <- resample_dist(x_unif)

# 3 Normal Distribution
x_norm    <-rnorm(n = n, mean = 0, sd = 1)
dist_norm <- resample_dist(x_norm)

# 4 Binomial Distribution
x_binom    <- rbinom(n = n, size = 500, prob = 0.7)
dist_binom <- resample_dist(x_binom)

# 5 Chisquare Distribution
x_chisq    <- rchisq(n = 4000, df = 10, ncp = 1)
dist_chisq <- resample_dist(x_binom)


# Bind data together
bind_rows( tibble(values = x_exp,      dist = "Exponential Distribution", var = "Histogram") )   |> 
bind_rows( tibble(values = dist_exp,   dist = "Exponential Distribution", var = "Histogram of Resampling Distribution") ) |> 
bind_rows( tibble(values = x_unif,     dist = "Uniform Distribution",     var = "Histogram") ) |> 
bind_rows( tibble(values = dist_unif,  dist = "Uniform Distribution",     var = "Histogram of Resampling Distribution") ) |> 
bind_rows( tibble(values = x_norm,     dist = "Normal Distribution",      var = "Histogram") ) |>
bind_rows( tibble(values = dist_norm,  dist = "Normal Distribution",      var = "Histogram of Resampling Distribution") ) |> 
bind_rows( tibble(values = x_binom,    dist = "Binomial Distribution",    var = "Histogram") ) |>
bind_rows( tibble(values = dist_binom, dist = "Binomial Distribution",    var = "Histogram of Resampling Distribution") ) |> 
bind_rows( tibble(values = x_chisq,    dist = "Chisquare Distribution",   var = "Histogram") ) |>
bind_rows( tibble(values = dist_chisq, dist = "Chisquare Distribution",   var = "Histogram of Resampling Distribution") ) |> 

# Change order
mutate(dist = dist |> forcats::fct_relevel("Exponential Distribution",
                                           "Uniform Distribution",
                                           "Normal Distribution",
                                           "Binomial Distribution")) |> 
# Plot
ggplot(aes(values)) + 
  geom_histogram() +
  facet_wrap(vars(dist, var), scales = "free", dir = "v", nrow = 2) +
  theme(axis.title = element_blank(),
        axis.text  = element_blank(),
        axis.ticks = element_blank())
```
https://audhiaprilliant.medium.com/the-statistics-simulation-of-central-limit-theorem-and-law-of-large-number-b4e489a139c4

> *Question*
> ---
> Try creating sampling distributions of different summary statistics from samples of different distributions. Which distribution does the central limit theorem not apply to?<br>
> <br>
> ⬜ Discrete uniform distribution<br>
> ⬜ Continuous uniform distribution<br>
> ⬜ Binomial distribution<br>
> ⬜ All of the above<br>
> ✅ None of the above<br>

Victorious visualizing! Regardless of the shape of the distribution you're taking sample means from, the central limit theorem will apply if the sampling distribution contains enough sample means.

## The CLT in action

The central limit theorem states that a sampling distribution of a sample statistic approaches the normal distribution as you take more samples, no matter the original distribution being sampled from.

In this exercise, you'll focus on the sample mean and see the central limit theorem in action while examining the `num_users` column of `amir_deals` more closely, which contains the number of people who intend to use the product Amir is selling.

**Steps**

1. Create a histogram of the `num_users` column of `amir_deals`. Use 10 bins.

```{r}
# 1. Create a histogram of num_users
ggplot(amir_deals, aes(num_users)) +
    geom_histogram(bins = 10)
```

2. Set the seed to `104`.
3. Take a sample of size `20` with replacement from the `num_users` column of `amir_deals`, and take the mean.

```{r}
# 2. Set seed to 104
set.seed(104)

# 3. Sample 20 num_users with replacement from amir_deals
sample(amir_deals$num_users, 20, replace = T) %>%
  # Take mean
  mean()
```

4. Repeat this 100 times and store as `sample_means`. This will take 100 different samples and calculate the mean of each.


```{r}
set.seed(104)
# 4. Repeat the above 100 times
sample_means <- replicate(100, sample(amir_deals$num_users, size = 20, replace = TRUE) %>% mean())
```

5. A data frame called `samples` has been created for you with a column `mean`, which contains the values from `sample_means`. Create a histogram of the `mean` column with 10 bins.

```{r}
# Set seed to 104
set.seed(104)


# Create data frame for plotting
samples <- data.frame(mean = sample_means)

# Histogram of sample means
ggplot(samples, aes(mean)) +
  geom_histogram(bins=10)
```

Fabulous job! You've just seen the central limit thorem at work. Even though the distribution of `num_users` is not normal, the distribution of its sample mean resembles the normal distribution.

## The mean of means

You want to know what the average number of users (`num_users`) is per deal, but you want to know this number for the entire company so that you can see if Amir's deals have more or fewer users than the company's average deal. The problem is that over the past year, the company has worked on more than ten thousand deals, so it's not realistic to compile all the data. Instead, you'll estimate the mean by taking several random samples of deals, since this is much easier than collecting data from everyone in the company.

The user data for all the company's deals is available in `all_deals`.

**Steps**

1. Set the random seed to `321`.
2. Take 30 samples of size 20 from `all_deals$num_users` and take the mean of each sample. Store the sample means in `sample_means`.
3. Take the mean of `sample_means`.

```{r}
all_deals <- readRDS("data/all_deals.rds")

# 1. Set seed to 321
set.seed(321)

# 2. Take 30 samples of 20 values of num_users, take mean of each sample
sample_means <- replicate(30, sample(x = all_deals$num_users, size = 20) %>% mean())

# 3. Calculate mean of sample_means
mean(sample_means)
```

4. Take the mean of the `num_users` column of `amir_deals`.

```{r}
# 4. Calculate mean of num_users in amir_deals
mean(amir_deals$num_users)
```

Magnificent mean calculation! Amir's average number of users is very close to the overall average, so it looks like he's meeting expectations. Make sure to note this in his performance review!

## The Poisson distribution

Theory. Coming soon ...


**1. The Poisson distribution**

In this video, we'll talk about another probability distribution called the Poisson distribution.

**2. Poisson processes**

Before we talk about probability, let's define a Poisson process.A Poisson process is a process where events appear to happen at a certain rate, but completely at random. For example, the number of animals adopted from an animal shelter each week is a Poisson process - we may know that on average there are 8 adoptions per week, but this number can differ randomly. Other examples would be the number of people arriving at a restaurant each hour, or the number of earthquakes per year in California.

**3. Poisson distribution**

The Poisson distribution describes the probability of some number of events happening over a fixed period of time. We can use the Poisson distribution to calculate the probability of at least 5 animals getting adopted in a week, the probability of 12 people arriving in a restaurant in an hour, or the probability of fewer than 20 earthquakes in California in a year.

**4. Lambda ($\lambda$)**

The Poisson distribution is described by a value called lambda, which represents the average number of events per time period. In the animal shelter example, this would be the average number of adoptions per week, which is 8. This value is also the expected value of the distribution!The Poisson distribution with lambda equals 8 looks like this. Notice that it's a discrete distribution since we're counting events, and 7 and 8 are the most likely number of adoptions to happen in a week.

**5. Lambda is the distribution's peak**

Lambda changes the shape of the distribution, so a Poisson distribution with lambda equals 1, in red, looks quite different than a Poisson distribution with lambda equals 8, in blue, but no matter what, the distribution's peak is always at its lambda value.

**6. Probability of a single value**

Given that the average number of adoptions per week is 8, what's the probability of 5 adoptions in a week? We'll use the dpois function, passing 5 as the first argument and 8 as the second argument to indicate the distribution's mean. This gives us about 9%.

**7. Probability of less than or equal to**

To get the probability that 5 or fewer adoptions will happen in a week, use the ppois function, passing in the same numbers. This gives us about 20%.

**8. Probability of greater than**

Just like other probability functions you've learned about so far, use the lower-dot-tail argument to get the probability of more than 5 adoptions. There's an 81% chance that more than 5 adoptions will occur.If the average number of adoptions rises to 10 per week, there will be a 93% chance that more than 5 adoptions will occur.

**9. Sampling from a Poisson distribution**

Just like other distributions, we can take samples from Poisson distributions using rpois. Here, we'll simulate 10 different weeks at the animal shelter. In one week, there are 13 adoptions, but only 3 in another.

**10. The CLT still applies!**

Just like other distributions, the sampling distribution of sample means of a Poisson distribution looks normal with a large number of samples.

**11. Let's practice!**

Time to practice taking Poisson probabilities!

## Identifying lambda

Now that you've learned about the Poisson distribution, you know that its shape is described by a value called lambda. In this exercise, you'll match histograms to lambda values.

```{r}
c(1,2,5,10,20) |> 
  set_names() |>
  map(rpois, n = 10000) |> 
  map(as_tibble) |> 
  bind_rows(.id = "lambda") |> 
  mutate(lambda = lambda |> forcats::fct_inorder()) |> 
  ggplot(aes(x=value)) + 
  geom_density(aes(group = lambda, color=lambda, fill=lambda), adjust = 4, alpha = 1/3) +
  scale_color_discrete() + 
  scale_fill_discrete() + 
  ggtitle("Probability Mass Function")
```

Match each Poisson distribution to its lambda value.

## Tracking lead responses

Your company uses sales software to keep track of new sales leads. It organizes them into a queue so that anyone can follow up on one when they have a bit of free time. Since the number of lead responses is a countable outcome over a period of time, this scenario corresponds to a Poisson distribution. On average, Amir responds to 4 leads each day. In this exercise, you'll calculate probabilities of Amir responding to different numbers of leads.

**Steps**

1. What's the probability that Amir responds to 5 leads in a day, given that he responds to an average of 4?

```{r}
# 1. Probability of 5 responses
dpois(5, lambda = 4)
```

2. Amir's coworker responds to an average of 5.5 leads per day. What is the probability that she answers 5 leads in a day?

```{r}
# 2. Probability of 5 responses from coworker
dpois(5, 5.5)
```

3. What's the probability that Amir responds to 2 or fewer leads in a day?

```{r}
# Probability of 2 or fewer responses
ppois(2, 4)
```

4. What's the probability that Amir responds to more than 10 leads in a day?

```{r}
# Probability of > 10 responses
ppois(10, 4, lower.tail =F)
```

Perfect Poisson probabilities! Note that if you provide `dpois()` or `ppois()` with a non-integer, it returns 0 and throws a warning since the Poisson distribution only applies to integers.

## More probability distributions

Theory. Coming soon ...


**1. More probability distributions**

In this lesson, we'll discuss a few other probability distributions.

**2. Exponential distribution**

The first distribution is the exponential distribution, which represents the probability of a certain time passing between Poisson events. We can use the exponential distribution to predict, for example, the probability of more than 1 day between adoptions, the probability of fewer than 10 minutes between restaurant arrivals, and the probability of 6-8 months passing between earthquakes. The exponential distribution uses the same lambda value, which represents the rate, that the Poisson distribution does. Note that lambda and rate mean the same value in this context.It's also continuous, unlike the Poisson distribution, since it represents time.

**3. Customer service requests**

For example, let's say that one customer service ticket is created every 2 minutes. We can rephrase this so it's in terms of a time interval of one minute, so half of a ticket is created each minute. We'll use 0-point-5 as the lambda value.The exponential distribution with a rate of one half looks like this.

**4. Lambda in exponential distribution**

The rate affects the shape of the distribution and how steeply it declines.

**5. How long until a new request is created?**

Similar to other continuous distributions, we can use pexp to calculate probabilities.The probability of waiting less than 1 minute for a new request is calculated using pexp, passing in 1 and 0-point-5 as the rate, which gives us about a 40% chance.The probability of waiting more than 4 minutes can be found using lower-dot-tail equals FALSE, giving a 13% chance.Finally, the probability of waiting between 1 and 4 minutes can be found by taking pexp of 4 and subtracting pexp of 1. There's a 50% chance you'll wait between 1 and 4 minutes.

**6. Expected value of exponential distribution**

Recall that lambda is the expected value of the Poisson distribution, which measures frequency in terms of rate or number of events. In our customer service ticket example, this means that the expected number of requests per minute is 0-point-5.The exponential distribution measures frequency in terms of time between events. The expected value of the exponential distribution can be calculated by taking 1 divided by lambda. In our example, the expected time between requests is 1 over one half, which is 2, so there is an average of 2 minutes between requests.

**7. (Student's) t-distribution**

The next distribution is the t-distribution, which is also sometimes called Student's t-distribution. Its shape is similar to the normal distribution, but not quite the same.If we compare the normal distribution, in red, with the t-distribution with one degree of freedom, in blue, the t-distribution's tails are thicker. This means that in a t-distribution, observations are more likely to fall further from the mean.

**8. Degrees of freedom**

The t-distribution has a parameter called degrees of freedom, which affects the thickness of the distribution's tails. Lower degrees of freedom results in thicker tails and a higher standard deviation. As the number of degrees of freedom increases, the distribution looks more and more like the normal distribution.

**9. Log-normal distribution**

The last distribution we'll discuss is the log-normal distribution. Variables that follow a log-normal distribution have a logarithm that is normally distributed.This results in distributions that are skewed, unlike the normal distribution.There are lots of real-world examples that follow this distribution, such as the length of chess games, blood pressure in adults, and the number of hospitalizations in the 2003 SARS outbreak.

**10. Let's practice!**

In addition to the three in this video, there are lots of other probability distributions that are out of the scope of this course, but that you can learn about in other DataCamp courses. For now, it's time to practice the distributions you've learned so far!

## Too many distributions

By this point, you've learned about so many different probability distributions that it can be difficult to remember which is which. In this exercise, you'll practice distinguishing between distributions and identifying the distribution that best matches different scenarios.

Match each situation to the distribution that best models it.

| Poisson | Exponential | Binomial |
| --- | --- | --- |
| Number of customers that enter a store each hour | Amount of time until the next customer makes a purchase | Number of people from a group of 30 that pass their driving test |
| Number of products sold each week | Amount of time until someone pays off their loan | |

## Modeling time between leads

To further evaluate Amir's performance, you want to know how much time it takes him to respond to a lead after he opens it. On average, it takes 2.5 hours for him to respond. In this exercise, you'll calculate probabilities of different amounts of time passing between Amir receiving a lead and sending a response.


**Steps**

1. What's the probability it takes Amir less than an hour to respond to a lead?

```{r}
# Probability response takes < 1 hour
pexp(1, rate = 0.4)
```

2. What's the probability it takes Amir more than 4 hours to respond to a lead?

```{r}
# Probability response takes > 4 hours
pexp(4, rate = 0.4, lower.tail = F)
```

3. What's the probability it takes Amir 3-4 hours to respond to a lead?

```{r}
# Probability response takes 3-4 hours
pexp(4, 0.4) - pexp(3, 0.4)
```

Excellent exponential computations! There's only about a 20% chance it will take Amir more than 4 hours to respond, so he's pretty speedy in his responses.

## The t-distribution

> *Question*
> ---
> Which statement is **not** true regarding the t-distribution?<br>
> <br>
> ⬜ The t-distribution has thicker tails than the normal distribution.<br>
> ⬜ A t-distribution with high degrees of freedom resembles the normal distribution.<br>
> ⬜ The number of degrees of freedom affects the distribution's variance.<br>
> ✅ The t-distribution is skewed.<br>

Terrific! The t-distribution is not skewed, just like the normal distribution, but it does have thicker tails and higher variance than the normal distribution.

# 4. Correlation and Experimental Design

In this chapter, you'll learn how to quantify the strength of a linear relationship between two variables, and explore how confounding variables can affect the relationship between two other variables. You'll also see how a study’s design can influence its results, change how the data should be analyzed, and potentially affect the reliability of your conclusions.

## Correlation

Theory. Coming soon ...


**1. Correlation**

Welcome to the final chapter of the course, where we'll talk about correlation and experimental design.

**2. Relationships between two variables**

Before we dive in, let's talk about relationships between numeric variables. We can visualize these kinds of relationships with scatterplots - in this scatterplot, we can see the relationship between the total amount of sleep mammals get and the amount of REM sleep they get.The variable on the x-axis is called the explanatory or independent variable, and the variable on the y-axis is called the response or dependent variable.

**3. Correlation coefficient**

We can also examine relationships between two numeric variables using a number called the correlation coefficient. This is a number between -1 and 1, where the magnitude corresponds to the strength of the relationship between the variables, and the sign, positive or negative, corresponds to the direction of the relationship.

**4. Magnitude = strength of relationship**

Here's a scatterplot of 2 variables, x and y, that have a correlation coefficient of 0-point-99. Since the data points are closely clustered around a line, we can describe this as a near-perfect or very strong relationship. If we know what x is, we'll have a pretty good idea of what the value of y could be.

**5. Magnitude = strength of relationship**

Here, x and y have a correlation coefficient of 0-point-75, and the data points are more spread out.

**6. Magnitude = strength of relationship**

In this plot, x and y have a correlation of 0-point-56 and are therefore moderately correlated.

**7. Magnitude = strength of relationship**

A correlation coefficient around 0-point-2 would be considered a weak relationship.

**8. Magnitude = strength of relationship**

When the correlation coefficient is close to 0, x and y have no relationship and the scatterplot looks completely random. This means that knowing the value of x doesn't tell us anything about the value of y.

**9. Sign = direction**

The sign of the correlation coefficient corresponds to the direction of the relationship. A positive correlation coefficient indicates that as x increases, y also increases. A negative correlation coefficient indicates that as x increases, y decreases.

**10. Visualizing relationships**

To visualize relationships between two variables, we can use a scatterplot created using geom_point.

**11. Adding a trendline**

We can add a linear trendline to the scatterplot using geom_smooth. We'll set the method argument to "lm" to indicate that we want a linear trendline, and se to FALSE so that there aren't error margins around the line.Trendlines like this can be helpful to more easily see a relationship between two variables.

**12. Computing correlation**

To calculate the correlation coefficient between two variables in R, we can use the cor function. The cor function takes in two numeric vectors and will return their correlation coefficient.Note that it doesn't matter which order the vectors are passed into the function since the correlation between x and y is the same thing as the correlation between y and x.

**13. Correlation with missing values**

If you have any missing values in either variable, R will return NA when you calculate correlation. To ignore data points where one or both values are missing, set the use argument of cor to pairwise-dot-complete-dot-obs.

**14. Many ways to calculate correlation**

There's more than one way to calculate correlation, but the method we've been using in this video is called the Pearson product-moment correlation, which is also written as r. This is the most commonly used measure of correlation. Mathematically, it's calculated using this formula where x and y bar are the means of x and y, and sigma x and sigma y are the standard deviations of x and y.The formula itself isn't important to memorize, but know that there are variations of this formula that measure correlation a bit differently, such as Kendall's tau and Spearman's rho, but those are beyond the scope of this course.

**15. Let's practice!**

Okay, time to practice calculating correlations.

## Guess the correlation

> *Question*
> ---
> Which of the following statements is NOT true about correlation?<br>
> <br>
> ⬜ If the correlation between `x` and `y` has a high magnitude, the data points will be clustered closely around a line.<br>
> ⬜ Correlation can be written as *r*.<br>
> ⬜ If `x` and `y` are negatively correlated, values of `y` decrease as values of `x` increase.<br>
> ✅ Correlation cannot be 0.<br>

Spot on! When correlation is 0, that means there is no relationship between two variables and the points appear to be randomly scattered.

## Relationships between variables

In this chapter, you'll be working with a dataset `world_happiness` containing results from the <a href="https://worldhappiness.report/ed/2019/" target="_blank" rel="noopener noreferrer">2019 World Happiness Report</a>. The report scores various countries based on how happy people in that country are. It also ranks each country on various societal aspects such as social support, freedom, corruption, and others. The dataset also includes the GDP per capita and life expectancy for each country.

In this exercise, you'll examine the relationship between a country's life expectancy (`life_exp`) and happiness score (`happiness_score`) both visually and quantitatively. Both `dplyr` and `ggplot2` are loaded and `world_happiness` is available.

**Steps**

1. Create a scatterplot of `happiness_score` vs. `life_exp` using `ggplot2`.

```{r}
world_happiness <- readRDS("data/world_happiness_sugar.rds")

# 1. Create a scatterplot of happiness_score vs. life_exp
ggplot(world_happiness, aes(x = life_exp, y = happiness_score)) +
    geom_point()
```

2. Add a linear trendline to the scatterplot, setting `se` to `FALSE`.

```{r}
# 2. Add a linear trendline to scatterplot
ggplot(world_happiness, aes(life_exp, happiness_score)) +
  geom_point() +
  geom_smooth(method = "lm", se = F)
```

> *Question*
> ---
> Based on the scatterplot, which is most likely the correlation between `life_exp` and `happiness_score`?<br>
> <br>
> ⬜ 0.3<br>
> ⬜ -0.3<br>
> ✅ 0.8<br>
> ⬜ -0.8<br>

3. Calculate the correlation between `life_exp` and `happiness_score`.

```{r}
# 3. Correlation between life_exp and happiness_score
cor(world_happiness$life_exp, world_happiness$happiness_score)
```
Vibrant visualizations! Scatterplots with trendlines are a great way to verify that a relationship between two variables is indeed linear.

## Correlation caveats

Theory. Coming soon ...


**1. Correlation caveats**

While correlation is a useful way to quantify relationships, there are some caveats.

**2. Non-linear relationships**

Consider this data. There is clearly a relationship between x and y, but when we calculate the correlation, we get 0-point-18.

**3. Non-linear relationships**

This is because the relationship between the two variables is a quadratic relationship, not a linear relationship. The correlation coefficient measures the strength of linear relationships, and linear relationships only.

**4. Correlation only accounts for linear relationships**

Just like any summary statistic, correlation shouldn't be used blindly, and you should always visualize your data when possible.

**5. Mammal sleep data**

Let's return to the mammal sleep data we discussed in chapter 1.

**6. Body weight vs. awake time**

Here's a scatterplot of each mammal's body weight versus the time they spend awake each day. The relationship between these variables is definitely not a linear one.The correlation between body weight and awake time is only about 0-point-3, which is a weak linear relationship.

**7. Distribution of body weight**

If we take a closer look at the distribution of body weight, it's highly skewed. There are lots of lower weights and a few weights that are much higher than the rest.

**8. Log transformation**

When data is highly skewed like this, we can apply a log transformation. We'll create a new column called log_bodywt which holds the log of each body weight.If we plot the log of bodyweight versus awake time, the relationship looks much more linear than the one between regular bodyweight and awake time.The correlation between the log of bodyweight and awake time is about 0-point-57, which is much higher than the 0-point-3 we had before.

**9. Other transformations**

In addition to the log transformation, there are lots of other transformations that can be used to make a relationship more linear, like taking the square root or reciprocal of a variable. The choice of transformation will depend on the data and how skewed it is. These can be applied in different combinations to x and y, for example, you could apply a log transformation to both x and y, or a square root transformation to x and a reciprocal transformation to y.

**10. Why use a transformation?**

So why use a transformation?Certain statistical methods rely on variables having a linear relationship, like calculating a correlation coefficient. Linear regression is another statistical technique that requires variables to be related in a linear manner, which you can learn all about in this course.

**11. Correlation does not imply causation**

Let's talk about another important caveat of correlation that you may have heard about before: correlation does not imply causation. This means that if x and y are correlated, x doesn't necessarily cause y.For example, here's a scatterplot of the per capita margarine consumption in the US each year versus the divorce rate in the state of Maine. The correlation between these two variables is 0-point-99, which is nearly perfect. However, this doesn't mean that consuming more margarine will cause more divorces. This kind of correlation is often called a spurious correlation.

**12. Confounding**

A phenomenon called confounding can lead to spurious correlations.Let's say we want to know if drinking coffee causes lung cancer. Looking at the data, we find that coffee drinking and lung cancer are correlated, which may lead us to think that drinking more coffee will give you lung cancer.

**13. Confounding**

However, there is a third, hidden variable at play, which is smoking.

**14. Confounding**

Smoking is known to be associated with coffee consumption.

**15. Confounding**

It is also known that smoking causes lung cancer.

**16. Confounding**

In reality, it turns out that coffee does not cause lung cancer and is only associated with it, but it appeared causal due to the third variable, smoking.This third variable is called a confounder, or lurking variable. This means that the relationship of interest between coffee and lung cancer is a spurious correlation.Another example of this is the relationship between holidays retail sales. While it might be that people buy more around holidays as a way of celebrating, it's hard to tell how much of the increased sales is due to holidays, and how much is due to the special deals and promotions that often run around holidays. Here, special deals confound the relationship between holidays and sales.

**17. Let's practice!**

Now that you've learned how to use correlation responsibly, time to practice.

## What can't correlation measure?

While the correlation coefficient is a convenient way to quantify the strength of a relationship between two variables, it's far from perfect. In this exercise, you'll explore one of the caveats of the correlation coefficient by examining the relationship between a country's GDP per capita (`gdp_per_cap`) and happiness score.

**Steps**

1. Create a scatterplot showing the relationship between `gdp_per_cap` (on the x-axis) and `life_exp` (on the y-axis).

```{r}
# 1. Scatterplot of gdp_per_cap and life_exp
ggplot(world_happiness, aes(x = gdp_per_cap, y = life_exp)) +
    geom_point()
```

2. Calculate the correlation between `gdp_per_cap` and `life_exp`.

```{r}
# Correlation between gdp_per_cap and life_exp
cor(world_happiness$gdp_per_cap, world_happiness$life_exp)
```

> *Question*
> ---
> The correlation between GDP per capita and life expectancy is 0.7. Why is correlation not the best way to measure the relationship between the two variables?<br>
> <br>
> ⬜ Correlation measures how one variable affects another.<br>
> ✅ Correlation only measures linear relationships.<br>
> ⬜ Correlation cannot properly measure relationships between numeric variables.<br>

Correct! The correlation coefficient can't account for any relationships that aren't linear, regardless of strength. 
  
## Transforming variables

When variables have skewed distributions, they often require a transformation in order to form a linear relationship with another variable so that correlation can be computed. In this exercise, you'll perform a transformation yourself.

**Steps**

1. Create a scatterplot of `happiness_score` versus `gdp_per_cap`.

```{r}
# 1. Scatterplot of happiness_score vs. gdp_per_cap
ggplot(world_happiness, aes(x=gdp_per_cap, y = happiness_score)) +
    geom_point()
```

2. Calculate the correlation between `happiness_score` and `gdp_per_cap`.

```{r}
# 2. Calculate correlation
cor(world_happiness$happiness_score, world_happiness$gdp_per_cap)
```

3. Add a new column to `world_happiness` called `log_gdp_per_cap` that contains the log of `gdp_per_cap`.
4. Create a scatterplot of `happiness_score` versus `log_gdp_per_cap`.

```{r}
# 3. Create log_gdp_per_cap column
world_happiness <- world_happiness %>%
  mutate(log_gdp_per_cap = log(gdp_per_cap))

# 4. Scatterplot of log_gdp_per_cap vs. happiness_score
ggplot(world_happiness, aes(x=log_gdp_per_cap, y=happiness_score)) +
  geom_point()
```

5. Calculate the correlation between `happiness_score` and `log_gdp_per_cap`.

```{r}
# 5. Calculate correlation
cor(world_happiness$log_gdp_per_cap, world_happiness$happiness_score)
```

Terrific transforming! The relationship between GDP per capita and happiness became more linear by applying a log transformation. Log transformations are great to use on variables with a skewed distribution, such as GDP.

## Does sugar improve happiness?

A new column has been added to `world_happiness` called `grams_sugar_per_day`, which contains the average amount of sugar eaten per person per day in each country. In this exercise, you'll examine the effect of a country's average sugar consumption on its happiness score.

**Steps**

1. Create a scatterplot showing the relationship between `grams_sugar_per_day` (on the x-axis) and `happiness_score` (on the y-axis).

```{r}
# Scatterplot of grams_sugar_per_day and happiness_score
ggplot(world_happiness, aes(x=grams_sugar_per_day, y = happiness_score)) +
    geom_point()
```

2. Calculate the correlation between `grams_sugar_per_day` and `happiness_score`.

```{r}
# Correlation between grams_sugar_per_day and happiness_score
cor(world_happiness$grams_sugar_per_day, world_happiness$happiness_score)
```

> *Question*
> ---
> Based on this data, which statement about sugar consumption and happiness scores is true?<br>
> <br>
> ⬜ Increased sugar consumption leads to a higher happiness score.<br>
> ⬜ Lower sugar consumption results in a lower happiness score<br>
> ✅ Increased sugar consumption is associated with a higher happiness score.<br>
> ⬜ Sugar consumption is not related to happiness.<br>

Nice interpretation of correlation! If correlation always implied that one thing causes another, people may do some nonsensical things, like eat more sugar to be happier. 

## Confounders

A study is investigating the relationship between neighborhood residence and lung capacity. Researchers measure the lung capacity of thirty people from neighborhood A, which is located near a highway, and thirty people from neighborhood B, which is not near a highway. Both groups have similar smoking habits and a similar gender breakdown. 

> *Question*
> ---
> Which of the following could be a *confounder* in this study?<br>
> <br>
> ⬜  Lung capacity<br>
> ⬜  Neighborhood<br>
> ✅  Air pollution<br>
> ⬜  Smoking status<br>
> ⬜  Gender<br>

Correct! You would expect there to be more air pollution in the neighborhood situated near the highway, which can cause lower lung capacity.

## Design of experiments

Theory. Coming soon ...

**1. Design of experiments**

Often, data is created as a result of a study that aims to answer a specific question. However, data needs to be analyzed and interpreted differently depending on how the data was generated and how the study was designed.

**2. Vocabulary**

Experiments generally aim to answer a question in the form, "What is the effect of the treatment on the response?" In this setting, treatment refers to the explanatory or independent variable, and response refers to the response or dependent variable. For example, what is the effect of an advertisement on the number of products purchased? In this case, the treatment is an advertisement, and the response is the number of products purchased.

**3. Controlled experiments**

In a controlled experiment, participants are randomly assigned to either the treatment group or the control group, where the treatment group receives the treatment and the control group does not. In our example, the treatment group will see an advertisement, and the control group will not. Other than this difference, the groups should be comparable so that we can determine if seeing an advertisement causes people to buy more. If the groups aren't comparable, this could lead to confounding, or bias. If the average age of participants in the treatment group is 25 and the average age of participants in the control group is 50, age could be a potential confounder if younger people are more likely to purchase more, and this will make the experiment biased towards the treatment.

**4. The gold standard of experiments will use...**

The gold standard, or ideal experiment, will eliminate as much bias as possible using certain tools. The first tool to help eliminate bias in controlled experiments is to use a randomized controlled trial. In a randomized controlled trial, participants are randomly assigned to the treatment or control group and their assignment isn't based on anything other than chance. Random assignment like this helps ensure that the groups are comparable. The second way is to use a placebo, which is something that resembles the treatment, but has no effect. This way, participants don't know if they're in the treatment or control group. This ensures that the effect of the treatment is due to the treatment itself, not the idea of getting the treatment. This is common in clinical trials that test the effectiveness of a drug. The control group will still be given a pill, but it's a sugar pill that has minimal effects on the response.

**5. The gold standard of experiments will use...**

In a double-blind experiment, the person administering the treatment or running the experiment also doesn't know whether they're administering the actual treatment or the placebo. This protects against bias in the response as well as the analysis of the results. These different tools all boil down to the same principle: if there are fewer opportunities for bias to creep into your experiment, the more reliably you can conclude whether the treatment affects the response.

**6. Observational studies**

The other kind of study we'll discuss is the observational study. In an observational study, participants are not randomly assigned to groups. Instead, participants assign themselves, usually based on pre-existing characteristics. This is useful for answering questions that aren't conducive to a controlled experiment. If you want to study the effect of smoking on cancer, you can't force people to start smoking. Similarly, if you want to study how past purchasing behavior affects whether someone will buy a product, you can't force people to have certain past purchasing behavior. Because assignment isn't random, there's no way to guarantee that the groups will be comparable in every aspect, so observational studies can't establish causation, only association. The effects of the treatment may be confounded by factors that got certain people into the control group and certain people into the treatment group. However, there are ways to control for confounders, which can help strengthen the reliability of conclusions about association.

**7. Longitudinal vs. cross-sectional studies**

The final important distinction to make is between longitudinal and cross-sectional studies. In a longitudinal study, the same participants are followed over a period of time to examine the effect of treatment on the response. In a cross-sectional study, data is collected from a single snapshot in time. If you wanted to investigate the effect of age on height, a cross-sectional study would measure the heights of people of different ages and compare them. However, the results will be confounded by birth year and lifestyle since it's possible that each generation is getting taller. In a longitudinal study,the same people would have their heights recorded at different points in their lives, so the confounding is eliminated. It's important to note that longitudinal studies are more expensive, and take longer to perform, while cross-sectional studies are cheaper, faster, and more convenient.

**8. Let's practice!**

Time to practice your study design skills! 

## Study types

While controlled experiments are ideal, many situations and research questions are not conducive to a controlled experiment. In a controlled experiment, causation can likely be inferred if the control and test groups have similar characteristics and don't have any systematic difference between them. On the other hand, causation cannot usually be inferred from observational studies, whose results are often misinterpreted as a result.

In this exercise, you'll practice distinguishing controlled experiments from observational studies.

Determine if each study is a controlled experiment or observational study.

| Controlled Experiment | Observational Study | 
| --- | --- |
| Purchasing rates are compared between users of an e-commerce site who are randomly directed to a new version of the home page or an old version. | A week ago, the home page of an e-commerce site was updated. Purchasing rates are compared between users who saw the old and new home page versions. |
| Subjects are randomly assigned to a diet and weight loss is compared. | Prevalence of heart disease is compared between veterans with PTSD and veterans without PTSD. |
| Asthma symptoms are compared between children randomly assigned to receive professional home pest management services or pest management education. | |

## Longitudinal vs. cross-sectional studies

<div class="">A company manufactures thermometers, and they want to study the relationship between a thermometer's age and its accuracy. To do this, they take a sample of 100 different thermometers of different ages and test how accurate they are. Is this data longitudinal or cross-sectional?

> *Question*
> ---
> Which of the following could be a *confounder* in this study?<br>
> <br>
> ⬜  Longitudinal<br>
> ✅  Cross-sectional<br>
> ⬜  Both<br>
> ⬜  Neither<br>

Perfect! This is a cross-sectional study since researchers aren't following the same set of thermometers over time and repeatedly measuring their accuracy at different ages.

## Congratulations!

Theory. Coming soon ...


**1. Congratulations!**

Congratulations on completing the course! You now have foundational statistics skills that you can use in your analyses and build upon further.

**2. Overview**

In the first chapter of the course, you learned about what statistics can do, as well as summary statistics to measure the center and spread of a distribution.In the second chapter, you learned how to measure chance and how to use and interpret probability distributions. You also learned about the binomial distribution.In chapter three, you learned about the normal distribution and the central limit theorem, one of the most important ideas in statistics. You also saw how the Poisson distribution can be used to model countable outcomes.In the final chapter, you saw how to quantify relationships between two variables using correlation. You also learned about controlled experiments and observational studies and the conclusions that can and cannot be drawn from them.

**3. Build on your skills**

There's still much more that you can do with statistics and much more to learn. Your new skills will set you up for success in this course on the foundations of regression.

**4. Congratulations!**

Thanks for accompanying me on this statistical journey. Congratulations again!