What is a Holdout Group? Definition and Examples in Analytics

Holdout group

A holdout group is a baseline subset in analytics experiments, excluded from changes to measure true incremental impact.

Introduction to Holdout Groups

• Definition

  A holdout group is a subset of your audience that does not receive the tested change, ensuring you have a clean baseline for comparison.

• Purpose in experiments

  The holdout group establishes what would have happened without the change, helping to compute the incremental lift attributable solely to your experiment.

How Holdout Groups Work

• Random assignment

  Users are typically assigned to the holdout or test group using a pseudorandom function, like JavaScript’s Math.random(), hashed user IDs, or built-in library features.

• Measuring incrementality

  Incrementality is calculated by comparing key metrics (e.g., conversions, engagement) between the test and holdout groups to isolate the effect of the change.

Implementing Holdout Groups in Analytics Tools

• Plainsignal implementation

  Use conditional script injection to exclude a percentage of visitors. For example, to create a 10% holdout group:

  <script>
    const isHoldout = Math.random() < 0.1; // 10% holdout
    if (!isHoldout) {
      const link = document.createElement('link');
      link.rel = 'preconnect';
      link.href = '//eu.plainsignal.com/';
      link.crossOrigin = '';
      document.head.appendChild(link);
      const script = document.createElement('script');
      script.defer = true;
      script.dataset.do = 'yourwebsitedomain.com';
      script.dataset.id = '0GQV1xmtzQQ';
      script.dataset.api = '//eu.plainsignal.com';
      script.src = '//cdn.plainsignal.com/PlainSignal-min.js';
      document.head.appendChild(script);
    } else {
      console.log('User in holdout group');
    }
  </script>
  

  • Randomize users:

    Use a pseudorandom function to assign each visitor consistently on page load.

  • Conditional loading:

    Load the PlainSignal script only for non-holdout users to exclude their data.

  • Analytics tagging:

    Optionally record the isHoldout flag in a custom dimension for deeper analysis.

• Ga4 implementation

  Filter GA4 hits by injecting configuration for non-holdout users. Example:

  <script>
    const isHoldout = Math.random() < 0.1;
    window.dataLayer = window.dataLayer || [];
    function gtag(){dataLayer.push(arguments);}
    if (!isHoldout) {
      gtag('js', new Date());
      gtag('config', 'GA_MEASUREMENT_ID');
    } else {
      console.log('User in holdout group');
    }
  </script>
  

  • Data layer setup:

    Initialize the dataLayer and gtag function before sending events.

  • Conditional config:

    Only call gtag('config') for users not in the holdout group.

  • Event filtering:

    Exclude or tag holdout users to avoid contaminating experiment data.

Best Practices

• Determine sample size

  Calculate the minimum required sample for statistical significance, considering expected effect size and variance.

• Set clear duration

  Run the experiment long enough to capture representative user behavior, accounting for weekly and monthly cycles.

• Maintain randomization

  Ensure the assignment algorithm remains consistent across sessions to prevent bucket hopping.

Common Pitfalls

• Segment overlap

  Avoid overlapping cohorts where users belong to multiple experiments or holdouts simultaneously.

• Insufficient sample

  Small holdout groups can lead to high variance and unreliable conclusions.

• Analysis bias

  Ensure post-experiment analysis accounts for invalidated sessions or tracking errors.

Related terms