TechDatadogabout 17 hours ago
Measuring cumulative experiment impact accurately
1 min read
Naive sums of winning experiment lifts overstate impact due to the winner's curse. Datadog's Cumulative Impact feature corrects estimates using empirical Bayes shrinkage, avoiding the need for a holdout. A holdout measures cumulative effects directly but requires advance planning and traffic allocation.
Level
Hype check
Tap to vote and see what everyone thinks.
