Don't trust vanity promo lifts: lightweight incrementality tests studios can actually run

Every studio owner has done this: you run a "$49 intro month" promo, watch 22 new sign-ups roll in, and call it a win. The promo "worked." You do it again next quarter.

The problem is that most of those numbers are lying to you. Not because the promo failed, but because you have no idea how many of those 22 people would have signed up without the discount. Some were already circling your booking page. Some had a friend dragging them in. Some found you through a Google search that had nothing to do with your promo. When you count all 22 as "promo-driven," you're inflating the lift and quietly training yourself to discount people who were going to pay full price anyway.

That gap between observed sign-ups and actual incremental sign-ups is the entire game. And for a small studio doing maybe 40–80 new leads a month, measuring it feels impossible — the volumes are too low for anything that sounds like real statistics. So most owners give up and trust the vanity number.

You don't have to. There are lightweight ways to measure promotion incrementality studio-side that work at small volumes, don't need a data scientist, and take maybe an hour to set up. This post walks through the designs that actually hold up, what you can realistically detect at studio scale, and templates you can drop into a spreadsheet this week.

Why "22 new members" tells you almost nothing

When you run a promo, three things happen at once:

1. 1
   Some people sign up because of the discount (true incremental).
2. 2
   Some people who were already going to sign up take the discount instead of paying full (cannibalized revenue).
3. 3
   Normal baseline demand keeps flowing regardless of the promo.

Your booking dashboard lumps all three into one number. It can't separate them because it never sees the counterfactual — the version of that week where you didn't run the promo.

A typical example: a studio runs a New Year promo and sees 30 intros. Feels huge. But January is their highest-demand month anyway. Their baseline January intro count over the previous two years was around 18–20. So the real incremental lift from the promo was closer to 10–12 sign-ups, not 30. And of those 30, maybe 8 would've bought a full-price intro at $75 instead of the $39 promo. That's roughly $290 in cannibalized revenue hiding inside a "successful" campaign.

The vanity number said +30. Reality was closer to +11, minus some margin damage. Those are two completely different business decisions.

The three lightweight designs that work at studio scale

You're not going to run a proper randomized A/B test with a control group of 5,000 users. You have low volume, seasonality, and a single location (or a few). These three designs are built for exactly that.

1. Week-over-week holdout (the simplest one)

You run the promo for a defined window, then compare it against a clean baseline period before the promo — ideally the same days of the week, adjusted for known seasonality.

1. 1
   Best for

   short 1–2 week promos with a clear on/off switch.

2. 2
   Watch out for

   seasonality. Comparing a September promo to a sleepy August baseline will overstate lift every time.

2. Time-block holdout (promo on / promo off, rotating)

Instead of running the promo continuously, you turn it on for a block, off for a block, on again. Two weeks on, two weeks off, two weeks on. Then you compare the "on" blocks against the "off" blocks within the same season.

1. 1
   Best for

   always-on style offers (referral bonuses, first-class-free) where you can toggle without confusing customers.

2. 2
   Watch out for

   anticipation. If regulars figure out "the promo comes back every other week," they wait. Rotate on an unpredictable schedule.

3. Simple cohort control (geography or segment split)

If you have two locations, or you can split your email list, you run the promo for one group and hold the other back as a control. Compare sign-up rates between them over the same window.

1. 1
   Best for

   multi-location studios or email-driven reactivation campaigns.

2. 2
   Watch out for

   the two groups have to be genuinely comparable. Splitting "downtown location vs suburban location" introduces differences that have nothing to do with the promo.

The difference in reactivation between comparable groups is the clearest signal you can get at studio scale — both halves experienced the same week, so seasonality drops out.

The uncomfortable truth about MDEs at studio volumes

MDE — minimum detectable effect — is the smallest lift your test can actually see given your sample size. At studio volumes, this is where reality bites.

If you're measuring promotion incrementality studio-wide with 50 new leads a month, you cannot reliably detect a 5% improvement. The noise is bigger than the signal. What you can detect is a big, obvious swing.

Here's a rough sense of what's realistic. These aren't lab-precise — they're the practical thresholds studios tend to run into:

The takeaway isn't "don't measure." It's stop chasing small effects you can't see. A promo that lifts sign-ups by 6% is invisible at your volume — you'll never separate it from random noise, and if you claim you detected it, you're fooling yourself. Design your promos to produce swings big enough that even a low-volume test can catch them.

Running the same promo across a longer window matters for the same reason. Two weeks of data at 50 leads/month is basically nothing. Accumulating the same design across a full quarter gives you enough events to say something honest. If you've already built a consistent marketing rhythm — and you should, ideally something like the structure in a 12-month marketing calendar that fills capacity — you can stack repeated measurements of the same promo type instead of judging each one in isolation.

A quick net-impact template you can build today

Vanity lift measures sign-ups. Net impact measures money you actually kept. Here's the calculation that matters, in order:

A quick visual of the calculation flow.

1. 1
   Observed sign-ups during promo — the raw number your dashboard shows.
2. 2
   Estimated baseline — expected sign-ups without the promo, from your comparable-period average.
3. 3
   Incremental sign-ups = observed − baseline.
4. 4
   Cannibalization estimate — of the incremental group, how many would've paid full price? Start with a conservative guess, like 20–30%, and refine over time.
5. 5
   Net incremental revenue = (incremental sign-ups × promo price) − (cannibalized sign-ups × price gap) − promo costs.

Worked example. A studio runs a $39 intro (normal price $79):

1. 1
   Observed intros

   26

2. 2
   Baseline for that period

   ~15

3. 3
   Incremental

   11

4. 4
   Assume 25% cannibalization → ~3 of those would've paid full
5. 5
   Cannibalized margin loss

   3 × ($79 − $39) = $120

6. 6
   Ad spend + admin

   ~$180

7. 7
   Net incremental value

   (11 × $39) − $120 − $180 = $129, plus 11 new people in your funnel

That $129 is a wildly different story than "we got 26 sign-ups!" — and it's the number that should decide whether you run it again. Sometimes the answer is yes because the downstream retention of those 11 people justifies a thin front-end margin. But now you're deciding with the real figure, not the vanity one.

A minimal checklist before you run any promo test

Before you launch anything, it's worth running through these quickly. Skipping even one of them — especially the tagging step — usually means you're stuck with a number you can't trust after the fact.

1. 1
   - [ ] Define the exact promo window (start/end dates, times) before launch.
2. 2
   - [ ] Pull a baseline from 3+ comparable prior periods, not just last week.
3. 3
   - [ ] Pick one design (holdout, time-block, or cohort) and commit to it.
4. 4
   - [ ] Confirm your expected effect is big enough to clear your MDE at your volume.
5. 5
   - [ ] Tag every promo sign-up cleanly so you can separate them later.
6. 6
   - [ ] Decide your cannibalization assumption up front — don't reverse-engineer it to make the promo look good.
7. 7
   - [ ] Log promo cost and admin time — they belong in net impact.

“

Tag promo sign-ups at checkout whenever possible — it saves hours of reconstruction later.

The tagging step is the one studios skip and regret. If promo sign-ups and organic sign-ups look identical in your booking system three months later, you can't reconstruct incrementality.

Most studio platforms let you attach a source or campaign tag at checkout — turning that on before the promo starts, not after, is the difference between a measurable test and a guess. It's also where having booking, source, and revenue data connected in one place actually pays off, because your baseline math depends on clean historical records you can trust.

When this makes sense — and when it doesn't

When it's worth doing: any promo you're planning to repeat. If you'll run intro offers every quarter, spending an hour measuring the first one honestly saves you from repeating a money-loser four times a year. Repeated promos are where incrementality testing compounds.

When it's a bad idea: one-time, tiny, or purely brand plays. If you're comping ten free classes for a charity event, don't build a holdout design around it. The measurement effort outweighs the decision it informs.

Who should hold off for now: studios doing under ~20 relevant events a month with no consistent baseline history. At that volume, almost nothing clears the MDE. Focus on building predictable demand first — getting your attendance patterns readable using something like simple attendance-forecasting methods — then start measuring incrementality once you have enough signal to work with.

Real scenario: a two-location studio kills a promo that "worked"

A small studio with two locations ran a recurring "bring-a-friend free week" promo. The dashboard loved it — around 40 new trials over the campaign. Everyone assumed it was their best channel.

They finally ran a cohort control: promo at Location A, nothing at Location B for the same three weeks. Location A pulled roughly 24 trials, Location B pulled about 19 on its own with zero promo. The incremental lift was closer to 5 trials, not 40. Once they subtracted cannibalized full-price friends and the staff time spent managing the free week, net impact was slightly negative. They dropped the promo, redirected the same effort into a targeted reactivation email to lapsed members, and measured that with a simple list split. The reactivation split showed a genuine, repeatable incremental lift — smaller headline number, real money behind it. The vanity promo had been eating margin for over a year while looking like a winner.

The bottom line worth remembering

Observed sign-ups are the easiest number to celebrate and the easiest to be fooled by. The studios that grow profitably aren't the ones running the most promos — they're the ones who know which promos actually added members versus which ones just handed discounts to people already walking through the door.

You don't need a big audience or a stats background to figure that out. A clean baseline, one of these three lightweight designs, an honest MDE check, and a net-impact calculation will tell you more than any dashboard lift ever will. Run it once on your next repeat promo. The number you get back might be smaller than you hoped — but it'll be the first one you can actually trust.

You don't need a big audience or a stats background to figure that out. A clean baseline, one of these three lightweight designs, an honest MDE check, and a net-impact calculation will tell you more than any dashboard lift ever will. Run it once on your next repeat promo.

The number you get back might be smaller than you hoped — but it'll be the first one you can actually trust.