What Happens When You Actually Test Reality

Most brands think growth works like this:

1. Cut what doesn’t work.
2. Scale what does.

Simple.

Clean.

Completely wrong.

The problem isn’t effort.

It’s attribution.

Most founders/ marketers trust what their dashboard tells them:

Meta shows ROAS.
Google shows conversions.
Email shows revenue.

So they optimise accordingly.

And slowly… they start breaking their own growth engine.

We’ve seen this enough times to know exactly what’s happening.

The issue isn’t that certain channels don’t work.

It’s that they don’t get credit.

So instead of debating it with founders…

we prove it.

With incrementality tests.

What is an incrementality holdout test?

It’s the closest thing you’ll get to truth in marketing.

You remove (or isolate) one variable…
keep everything else constant…

…and observe what actually changes.

Not what gets attributed.
What actually changes.

Experiment 1: YouTube in Maharashtra

This brand was doing:

• ₹3.5Cr/month on Shopify
• ₹1Cr+/month in ad spend
• YouTube ROAS: <0.5

On any dashboard, YouTube looks like a bad decision.

Which is exactly why most brands cut it.

We didn’t.

Because we knew what was happening.

YouTube wasn’t converting.

It was creating demand.

The problem was:

We needed to prove it.

Not to ourselves.

To the founder/management

So we designed a test.

We increased YouTube spend…

But only in one state.

Maharashtra.

• Total YouTube budget: ₹10L/month
• Maharashtra: ₹1.5L → ₹3L

Everything else remained unchanged.

No changes to Meta.
No changes to Google.
No changes to email.

One variable.

Before the test:

• Blended ROAS (MH): ~3.2
• Best month ever: 3.5
• M3 retention: ~12%

Then we waited.

6 months.

Here’s what happened: (MH only)

Month 1: 3.2
Month 2: 3.2
Month 3: 3.5
Month 4: 3.6
Month 5: 3.6
Month 6: 3.6

• ₹6–7L incremental revenue/month (from Maharashtra alone)
• M3 retention: 12% → 14% (first time in brand history)

We increased spend on a channel showing <0.5 ROAS…

…and the business became more profitable.

Experiment 2: Email (the “3% channel”)

This one is even more dangerous.

Because this time, the data looks clean.

Google Analytics showed email contributing <3% of revenue.

A new D2C lead joined the brand.

First instinct:

“Let’s shut email down.”

Again - we knew what was happening.

Email wasn’t being credited.

But it was driving behaviour.

So instead of debating…

we tested it.

We stopped all email communication in 3 states:

• Madhya Pradesh
• Rajasthan
• Gujarat

These states contributed ~18% of revenue.

No changes to ads.
No changes to anything else.

3 months later:

• Contribution dropped: 18% → 11%
• nCAC increased by 27%
  
  (almost evenly in all states +/-15%)

Same business.

Only difference?

No email.

The pattern

Different channels.

Same truth.

Channel

What attribution says

What actually happens

YouTube

<0.5 ROAS, Drives demand + revenue

Email

<3% revenue, Impacts CAC + conversion

What most founders don’t understand

Attribution tools answer:

Who got the credit?

But growth comes from:

What actually caused the outcome?

And those are not the same thing.

Why this matters

If you optimise for attribution:

• You cut demand creation
• You over-invest in capture
• You slow down growth

If you optimise for reality:

• You build demand
• You reduce CAC
• You compound growth

How to run this yourself

You don’t need complexity.

You need conviction.

• Pick one variable (channel, geo, audience)
• Change only that
• Keep everything else constant
• Let it run long enough
• Measure business outcomes

The cost of truth

These tests cost money.

You will:

• Spend more on something that “looks bad”
• Turn off something that feels risky

But not doing this costs more.

Because then you build your business on the wrong signals.

And at scale…

that breaks everything.

If you still rely on ROAS per ad or per channel to make decisions…

you’re not optimising your business.

You’re optimising your attribution.

‍