ADBE

Case Study

Joschka Schwarz

Hamburg University of Technology

Sunday, 30^th of March 2025

Table of Contents

Problem Statement

What are we solving for?

Optimizing Precision: Strategies for Minimizing Error Rates in VIP Waiter Robots

Scenario

• Voice-controlled waiter robots were developed to equip VIP aircraft, but they occasionally produce errors during operation.
• A software update was developed to reduce the frequency of errors.
• Test series were performed to verify the effectiveness of the update.

Objectives

1. Does the new software version reduce the error rate?
2. If yes, by how much?
3. Each error that occurs generates costs of 3,14 €, rolling out the software update 42k €. Which software version should be used?

Steps

Understanding what needs to be done

Navigating Excellence: Step-by-Step Solutions for Enhancing Accuracy

• General
• Specific (Step 1)
• Specific (Step 2)

1. Step: Error Rate Reduction
• Calculate Error Rates
• Perform Statistical Test
• Quantify Result
2. Step: Cost Analysis (Break-Even)
• Cost of Errors
• Cost of Software Update
• Compare Total Costs
3. Step: Conclusion

Step 1: Error Rate Reduction

Hypothesis:

• Null Hypothesis (\(H_0\)): The new software version does not reduce the error rate.
• Alternative Hypothesis (\(H_1\)): The new software version reduces the error rate.

1. Calculate Error Rates
• Determine the error rate for the old software version and the error rate for the new software version. Error rate can be calculated as the number of errors divided by the total number of operations.
2. Perform Statistical Test
• Use an appropriate statistical test (C.I. for the Difference in Proportions) to compare the error rates of the old and new software versions.
3. Quantify Confidence
• Quantify confidence by providing confidence intervals for the reduction in error rate if the null hypothesis is rejected.

Step 2: Cost Analysis (Break-Even)

1. Cost of Errors: Determine the total cost of errors for the old software version and the new software version.

\[Cost \: of \: Errors = Error \: Rate × Cost \: per \: Error × Number \: of \: Operations\]

2. Cost of Software Update: Add the cost of rolling out the software update to the total cost for the new software version.

\[Total \: Cost \: for \: New \: Software \: Version = Cost \: of \: Errors \: (New \: Software) + Cost \: of \: Software \: Update\]

3. Compare Total Costs: Compare the total cost of errors and the cost of software update for the old and new software versions.

Data sources

What is Data Integrity and why does it matter?

Test series were performed to verify the effectiveness of the update

• General
• Specific
• Code (1)
• Code (2)

data_joined_cleaned_tbl <- read_rds("01_data/02_wrangled/data_joined_cleaned.rds")
data_joined_cleaned_tbl

#> # A tibble: 2,256 × 3
#>    software_version id                                                   failure
#>               <dbl> <chr>                                                  <dbl>
#>  1              1.1 76e899853d32693594596d20837f7fbd681864daff696baacd2…       0
#>  2              1.1 3714cb3b078089ac23bd92ab34c184ae6b69e7389c772e9b639…       0
#>  3              1.1 8601f3d8df88259dd4d0f84a7d0a8b287caf5e08f52b9ca98e4…       0
#>  4              1.2 4b457e08955d94130bf2f90ff905bc76fb1db3942688ff3e228…       0
#>  5              1.1 8dbd0947776dbab34f635daaebabc38309d09f45118da420b30…       0
#>  6              1.2 d7241ff8b59eefc600c21a9f16c12d5045a193f546c22181380…       0
#>  7              1.2 1a42a494855da0ad19f0535740ee4a8d08c6073bdb9aec157af…       0
#>  8              1.1 419c2d6bdc8497b2adc6f1f645689d6094d1f077543448d3af9…       0
#>  9              1.2 8739b5739e334cf30fb624fb970ad575a7475ffbd479ec2aca1…       0
#> 10              1.1 4c76f9572167ccb21c2e0bc93f6be7cf2cd367a116b84baec79…       0
#> # ℹ 2,246 more rows

data_joined_cleaned_tbl |> 
   count(failure, software_version)

#> # A tibble: 4 × 3
#>   failure software_version     n
#>     <dbl>            <dbl> <int>
#> 1       0              1.1  1461
#> 2       0              1.2   736
#> 3       1              1.1    43
#> 4       1              1.2    16

# 1. Read data
device_sv_tbl   <- read_delim("01_data/01_raw/device_sv.csv",   delim = "|", col_types = "dc")
test_result_tbl <- read_delim("01_data/01_raw/test_result.csv", delim = "|", col_types = "dc")

# 2. Analyze data (I)
device_sv_tbl   |> get_dupes(id)

#> # A tibble: 2 × 3
#>   id                                                 dupe_count software_version
#>   <chr>                                                   <int>            <dbl>
#> 1 bc62efbd55c35066c7dd02b34009fd3c6075834f98eb20aef…          2              1.2
#> 2 bc62efbd55c35066c7dd02b34009fd3c6075834f98eb20aef…          2              1.1

test_result_tbl |> get_dupes(pid)

#> # A tibble: 2 × 3
#>   pid                                                      dupe_count failure
#>   <chr>                                                         <int>   <dbl>
#> 1 bc62efbd55c35066c7dd02b34009fd3c6075834f98eb20aef8f43e6e          2       0
#> 2 bc62efbd55c35066c7dd02b34009fd3c6075834f98eb20aef8f43e6e          2       0

# 3. Clean data (I)
device_sv_cleaned_tbl   <- device_sv_tbl   |> filter(id  != "bc62efbd55c35066c7dd02b34009fd3c6075834f98eb20aef8f43e6e")
test_result_cleaned_tbl <- test_result_tbl |> filter(pid != "bc62efbd55c35066c7dd02b34009fd3c6075834f98eb20aef8f43e6e")

# 4. Merge data
data_joined_tbl <- device_sv_cleaned_tbl |> 
                      left_join(test_result_cleaned_tbl, by = join_by(id == pid))

# 5. Analyze data (II)
data_joined_tbl |> count(software_version, failure)

#> # A tibble: 8 × 3
#>   software_version failure     n
#>              <dbl>   <dbl> <int>
#> 1              1.1       0  1461
#> 2              1.1       1    43
#> 3              1.1     100     1
#> 4              1.1      NA     1
#> 5              1.2      -1     1
#> 6              1.2       0   736
#> 7              1.2       1    16
#> 8              1.2      NA     2

# 3. Clean data (II)
data_joined_cleaned_tbl <- data_joined_tbl |> filter(failure %in% c(0,1))

Analysis

Turning raw data into actionable insights

Achieving Precision: 25% Reduction in Mean Error Rates for Waiter Robots

• General
• Specific

data_joined_cleaned_tbl |>
  summarise(
    error_rate = mean(failure),
    n          = n(),
    .by = software_version
  )

#> # A tibble: 2 × 3
#>   software_version error_rate     n
#>              <dbl> <chr>      <int>
#> 1              1.1 2.86%       1504
#> 2              1.2 2.13%        752

Warning

There is a 95% chance that the confidence interval of [-0,6 pp; 2,1 pp] contains the true difference in the proportion of error rates between the two software versions. In other words, we cannot say with 95% certainty that there is a difference in mean error rates between these versions.

• 90% Confidence Interval: [-0,4 pp; 1,8 pp]
• 95% Confidence Interval: [-0,6 pp, 2,1 pp]
• 99% Confidence Interval: [-1,0 pp, 2,5 pp]

\(Confidence \: interval = (p_1–p_2) ± z*\sqrt{\frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2}}\)

propCI(n1 = 1504, n2 = 752, p1 = 0.02859043, p2 = 0.02127660, alpha = 0.10, digits = 3) |> pluck("result") |> select(CI)

#> # A tibble: 1 × 1
#>   CI                        
#>   <chr>                     
#> 1 0.007 [90CI -0.004; 0.018]

propCI(n1 = 1504, n2 = 752, p1 = 0.02859043, p2 = 0.02127660, alpha = 0.05, digits = 3) |> pluck("result") |> select(CI)

#> # A tibble: 1 × 1
#>   CI                        
#>   <chr>                     
#> 1 0.007 [95CI -0.006; 0.021]

propCI(n1 = 1504, n2 = 752, p1 = 0.02859043, p2 = 0.02127660, alpha = 0.01, digits = 3) |> pluck("result") |> select(CI)

#> # A tibble: 1 × 1
#>   CI                        
#>   <chr>                     
#> 1 0.007 [99CI -0.010; 0.025]

Crossing the Threshold at 1,7M - Unveiling the Cost Efficiency of the Update

• General
• Code (Calc)
• Code (Plot)

Assumption

The law of large numbers: The average of the results obtained from a large number of independent identical trials should be close to the expected value and tends to become closer to the expected value as more trials are performed.

# Set unitary cost
cost_err_1.1  <- 0.02859043 * 3.14
cost_err_1.2  <- 0.0212766  * 3.14
cost_rout_1.2 <- 40000

# Create units column
units <- seq(from = 0, to = 3000000, by = 100000)

# Create total cost columns
total_cost_1.1 <- cost_err_1.1 * units
total_cost_1.2 <- cost_err_1.2 * units + cost_rout_1.2

# Build data frame
df <- data.frame(units, total_cost_1.1, total_cost_1.2)

# Calculate contribution margin & break even units
contribution_margin  <- cost_err_1.1 - cost_err_1.2
break_even_units     <- cost_rout_1.2 / contribution_margin 
break_even_units_chr <- scales::label_number(accuracy  = 0.1, 
                                             scale_cut = scales::cut_short_scale(),
                                             big.mark  = ".", decimal.mark = ",")(break_even_units)

ggplot(data = df, aes(x = units)) +
  geom_line(aes(y = cost_rout_1.2,
                col = "Rollout Cost")) +
  geom_line(aes(y = total_cost_1.1,
                col = "Total Cost v1.1")) +
  geom_line(aes(y = total_cost_1.2,
                col = "Total Cost v1.2")) +
  geom_segment(aes(x = break_even_units, xend = break_even_units, 
                   y = 0, yend = break_even_units*cost_err_1.1),
               linetype = "dashed", color = "#9B9B9B") +
  geom_segment(aes(x = 0, xend = break_even_units, 
                   y = break_even_units*cost_err_1.1, yend = break_even_units*cost_err_1.1),
                   linetype = "dashed", color = "#9B9B9B") +
  geom_point(aes(x = break_even_units, 
                 y = break_even_units*cost_err_1.1), 
             colour = "#9B9B9B", size = 4) +
  annotate("label", x = break_even_units, y = 0, label = paste("Break Even Point:", break_even_units_chr)) +
  scale_color_manual(labels = c("Rollout Cost", "Total Cost v1.1", "Total Cost v1.2"),
                     values = c("Rollout Cost" = "black", "Total Cost v1.1" = "#FFAD00", "Total Cost v1.2" = "#0A1D3D")) +
  scale_y_continuous(breaks = c(0, 40000, 100000, 200000),
                     labels = scales::dollar_format(suffix = " €", prefix = "", big.mark = ".", decimal.mark = ",")) +
  scale_x_continuous(labels = scales::unit_format(unit = "M", scale = 1e-6)) +
  geom_ribbon(data = df[df$total_cost_1.2 >= df$total_cost_1.1, ], aes(x = units, ymin = total_cost_1.2, ymax = total_cost_1.1), fill = "#0A1D3D", alpha = 0.15) +
  geom_ribbon(data = df[df$total_cost_1.2 <= df$total_cost_1.1, ], aes(x = units, ymin = total_cost_1.1, ymax = total_cost_1.2), fill = "#FFAD00", alpha = 0.15) +
  labs(title = "Break Even Point Graph",
       x = "Number of operations (in millions)",
       y = "Expected cost",
       color = NULL) +
  tidyquant::theme_tq()

Conclusion

What conclusion can you draw from this?

Striving for Perfection: Key Takeaways in Advancing Voice-Controlled Waiter Robots

Summary

1. Does the new software version reduce the error rate?
• In all likelihood (More tests needed, experimentation with other test statistics, …)
2. By how much?
• 25,6% / 0,73pp (taking our assumptions into account)
3. Which software version should be used?
• The updated version 1.2 (depending on type of operations and cost of errors)

Next Steps

• Better understanding of errors & cost of errors -> Maximizing „utility“ vs. minimizing expected cost
• Good practices for Data Integrity
• Looking ahead to version 1.3: Improving quality by analysis of process variables (combating the causes)