Comparing means

Laboratory of Statistics and Mathematics 2025/2026

Giuseppe Alfonzetti

Types of t-tests

One-sample t-test : Compare one mean vs a known/target value

Two-sample t-test: Compare means of two independent groups

Paired t-test: Compare means on paired data (e.g. same people, before and after treatment)

Case study A

Your company is running a marketing campaign and you are monitoring the performance of the landing page in terms of time-on-page (i.e. seconds spent by the user on the landing page) to assess its effectiveness.

Your direct competitor reached a time-on-page equal to \(10\) seconds.

You want to assess if your campaign performance is significantly different from your competitor benchmark.

Data

landing_data

# A tibble: 150 × 3
   user_id version           time_on_page
   <chr>   <chr>                    <dbl>
 1 user_2  standard_campaign        11.7 
 2 user_6  standard_campaign         8.25
 3 user_7  standard_campaign         9.74
 4 user_8  standard_campaign        10.4 
 5 user_10 standard_campaign         9.73
 6 user_11 standard_campaign         8.78
 7 user_13 standard_campaign         8.73
 8 user_15 standard_campaign        10.9 
 9 user_16 standard_campaign         7.49
10 user_19 standard_campaign         9.81
# ℹ 140 more rows

Plot

landing_data |>
  ggplot(aes(x = version, y = time_on_page)) +
    geom_point() +
    labs(x="", y="Time on page (s)")

Plot

landing_data |>
  ggplot(aes(x = version, y = time_on_page)) +
    geom_point() +
    stat_summary(fun = "mean", geom = "point", size = 4, color="red")+
    labs(x="", y="Time on page (s)")

Plot

landing_data |>
  ggplot(aes(x = version, y = time_on_page)) +
    geom_boxplot() +
    labs(x="", y="Time on page (s)")

Business question

Is the marketing campaign performing significantly different from the competitor benchmark, in terms of time-on-page?

landing_data |>
  group_by(version) |>
  summarise(
    n = n(),
    mean_time = mean(time_on_page),
    sd_time = sd(time_on_page)
  )

# A tibble: 1 × 4
  version               n mean_time sd_time
  <chr>             <int>     <dbl>   <dbl>
1 standard_campaign   150      9.78   0.950

We can compare the average time-on-page to the benchmark with a t-test, using the t.test()function

t.test(
  x=landing_data$time_on_page, 
  mu=10, 
  alternative = "two.sided"
  )


    One Sample t-test

data:  landing_data$time_on_page
t = -2.8933, df = 149, p-value = 0.004385
alternative hypothesis: true mean is not equal to 10
95 percent confidence interval:
 9.622404 9.928871
sample estimates:
mean of x 
 9.775637

Case study B

You assessed that the data highlight a small underperformance of your company campaign compared to the competitor one.

The design team provided you with an alternative version of the landing page.

You are interested to check if the modified landing page performs differently from the old one.

Case study B - Experiments

Now you can choose between:

Bring to the new landing pege a new sample of users to monitor the time-on-page;

Bring to the new landing page the same users who visited the old one.

In the first case, you can compare the average time-on-page of the standard and modified landing pages using a Two sample t-test. In the second case, you have to rely on a Paired t-test.

Data

modified_landing_data

# A tibble: 150 × 3
   user_id version           time_on_page
   <chr>   <chr>                    <dbl>
 1 user_1  modified_campaign        12.2 
 2 user_3  modified_campaign        11.3 
 3 user_4  modified_campaign        13.5 
 4 user_5  modified_campaign        15.7 
 5 user_9  modified_campaign        16.3 
 6 user_12 modified_campaign         9.63
 7 user_14 modified_campaign        10.8 
 8 user_17 modified_campaign        14.6 
 9 user_18 modified_campaign        11.2 
10 user_22 modified_campaign         9.73
# ℹ 140 more rows

full_data <- landing_data |> 
  bind_rows(modified_landing_data)
full_data

# A tibble: 300 × 3
   user_id version           time_on_page
   <chr>   <chr>                    <dbl>
 1 user_2  standard_campaign        11.7 
 2 user_6  standard_campaign         8.25
 3 user_7  standard_campaign         9.74
 4 user_8  standard_campaign        10.4 
 5 user_10 standard_campaign         9.73
 6 user_11 standard_campaign         8.78
 7 user_13 standard_campaign         8.73
 8 user_15 standard_campaign        10.9 
 9 user_16 standard_campaign         7.49
10 user_19 standard_campaign         9.81
# ℹ 290 more rows

Plot two groups

full_data |>
  ggplot(aes(x = version, y = time_on_page)) +
    geom_boxplot() +
    labs(x="", y="Time to event (s)")

Business question

Is the modified landing page performing significantly different from the old one, in terms of time-on-page?

full_data |>
  group_by(version) |>
  summarise(
    n = n(),
    mean_time = mean(time_on_page),
    sd_time = sd(time_on_page)
  )

# A tibble: 2 × 4
  version               n mean_time sd_time
  <chr>             <int>     <dbl>   <dbl>
1 modified_campaign   150     12.2    1.88 
2 standard_campaign   150      9.78   0.950

We can compare the average time-on-page between standard and modified landing pages, using the t.test()function

t.test(time_on_page ~ version, data = full_data, paired=FALSE)


    Welch Two Sample t-test

data:  time_on_page by version
t = 14.003, df = 220.23, p-value < 2.2e-16
alternative hypothesis: true difference in means between group modified_campaign and group standard_campaign is not equal to 0
95 percent confidence interval:
 2.071554 2.750155
sample estimates:
mean in group modified_campaign mean in group standard_campaign 
                      12.186491                        9.775637