Skip to content

Guides · Testing

How to run a geo holdout test

Marxx · Updated 2 Nov 2026 · 3 min read

How to run a geo holdout test: Pause ads in a few matched states, keep them on elsewhere and compare orders. Six steps to run a geo holdout test, a worked rupee example and the mistakes to avoid.

The short answer

To run a geo holdout test, keep your ads on in most regions and pause them in a few matched ones for three to four weeks. Then compare orders in both groups, using your own store data.

The extra orders in the regions that kept ads are what your ads caused. A geo holdout test answers one question: how many sales would I lose if I stopped these ads?

At a glance: Step 1: choose the metric, Step 2: choose matched regions, Step 3: set locations in Ads Manager, Step 4: run it clean
At a glance

Step 1: choose the metric

Pick orders or revenue per region from your own store or CRM, not from Meta. Meta credits sales to ads. Your store data shows what happened.

Step 2: choose matched regions

Take the last 8 to 12 weeks of orders by state. Put states with similar size and trend into two groups, a test group that keeps ads and a holdout group that pauses them. Use states that sell enough each week to show a change of a few per cent. Avoid splitting a single city, since people cross the line.

Step 3: set locations in Ads Manager

At the ad set level, go to Audience, then Locations. Add only the test states as locations, or add the whole country and exclude the holdout states (checked 2026-10-20, Meta help). Meta says who sees an ad depends on current and home location. A person who lives in a shown state may see your ad while visiting a held-out one. Meta also says accuracy cannot be guaranteed and some impressions can land outside your location settings. Expect a little leakage and note it.

Step 4: run it clean

  • Run for three to four whole weeks.
  • Change nothing else. No new offers or price cuts.
  • Keep the budget in test states steady.
  • Log any event that hits one group, such as a local festival or a stock-out.

Step 5: measure

Use the growth of the test group as the guide for what the holdout would have done. Expected holdout orders without the pause equal their old orders times the test group's growth. The gap between expected and actual is your lift.

Step 6: turn it into a decision

Multiply the extra orders by your average order value, then divide by the ad spend you paused. Compare the result with your break-even ROAS.

A worked example

Worked example

Say you run ads in 12 states. You pick 8 as the test group and 4 as the holdout, matched on 10 weeks of orders. For four weeks you pause ads in the 4 holdout states. Average order value is ₹1,500.

Weekly orders before Weekly orders during Change
Test states (ads on) 1,200 1,230 Up 2.5%
Holdout states (ads paused) 600 540 Down 10%
  • Expected holdout orders at the test group's growth: 600 x 1.025 = 615.
  • Orders lost to the pause: 615 minus 540 = 75 a week, so 300 over four weeks.
  • Sales lost: 300 x ₹1,500 = ₹4,50,000.
  • Ad spend you paused: ₹2,00,000.
  • Incremental return: ₹4,50,000 ÷ ₹2,00,000 = 2.25.

If your gross margin is 40%, break-even ROAS is 1 ÷ 0.40 = 2.5. At 2.25, ads in those states pay back less than they cost at the margin. Before you cut, repeat the test and look at repeat purchases. Say Meta had reported 3.5 in those states. The test shows how far platform credit and real lift can differ.

Tools

Meta's open-source GeoLift package is written in R under the MIT licence. Its readme says it helps pick test markets with power calculators and measures lift with synthetic control methods (checked 2026-10-20). Paid tools are listed in the best incrementality testing tools. For a lighter route, see how to measure incrementality on a small budget.

Mistakes to avoid

  • Choosing regions by gut, not by past sales
  • Running one week, which is mostly noise
  • Starting a sale in only one group
  • Reading Meta's numbers instead of your store's
  • Treating one test as final

FAQ

How many regions do I need?

At least two or three in the holdout group, and more where you can. One region is hard to trust, because a local event can swamp the result.

How long should the test run?

Three to four whole weeks. Longer helps if your buyers take time to decide.

Is a geo holdout test the same as a geo-lift test?

They are close. A geo-lift test usually adds statistical modelling to pick regions and judge the gap. The basic idea is the same.

See what your competitors are running today.

Marxx tracks every ad in your sector, shows you what is working and builds the next ad on the same canvas you publish from.