Creative testing at scale on Meta: a 2026 framework

Andromeda rewards distinct ideas, not twenty versions of the same one. Here is how to test concepts, how much budget each test needs, and when to kill.

A hand-painted magic show advertisement on a wall in Chennai, with bold lettering and a painted figure
Photo: McKay Savage from London, UK, CC BY 2.0, via Wikimedia Commons

A creative testing framework for Meta in 2026 tests distinct concepts (a different promise, person, format or proof), not small tweaks, because Meta's Andromeda retrieval system matches people to ads and near-identical ads give it no new options. Run a steady pipeline: a fixed share of budget for tests, enough spend per concept to see 25 or more conversions, clear kill rules, and graduation of winners into the main campaign. The research on online experiments supports testing more ideas with leaner tests, because a small share of ideas produces most of the gains.

Key takeaways
  • Test concepts (angle, persona, format, proof), not tweaks like colour or button text.
  • Give each concept about 25 conversions: noise is roughly 1/√n, so ±20% at 25.
  • At Bing, the top 2% of ideas produced 74.8% of gains, so test more ideas, more leanly.
  • Keep 15-25% of spend in a separate testing campaign with written kill rules.
  • Meta's ad-level numbers are attributed, not incremental: use them to rank, not to prove.

This post is for teams already spending ₹10 lakh or more a month on Meta who feel their creative is "burning out" faster than it used to.

Usually the creative is not the problem. The testing system is:

  • too many near-duplicates;
  • too little spend per idea;
  • no rule for what counts as a winner.

Why did creative testing on Meta change after Andromeda?

Creative testing changed because Meta's delivery now picks ads for each person from a huge pool, and the creative itself does much of the targeting that interest and demographic settings used to do.

Meta's engineering team described Andromeda in December 2024 as the retrieval stage of its ads system: it narrows tens of millions of candidate ads to a few thousand for the ranking models. Meta reported a 10,000x increase in model capacity for retrieval, a 6% gain in recall and an 8% gain in ads quality on selected segments, running on NVIDIA Grace Hopper hardware and Meta's own MTIA chips.

Two consequences matter for a media buyer:

  • Variety is the input. Jon Loomer quotes Meta's official line that creative diversification helps advertisers get the best results, and Meta's own April 2025 messaging spoke of a shift from niche targeting to creative diversification. Meta also dropped its old guidance of six ads per ad set; Loomer reports running ad sets with 50.
  • Similar ads add little. If ten ads share the same hook, the same face and the same offer, the retrieval system is choosing among ten copies of one idea. My reading, consistent with what Loomer advises, is that visuals need to be "distinctly different" for variety to count.

The deeper mechanics are in how Andromeda and Advantage+ work. This post is about what to do with them, starting with why nobody, however senior, can skip the test.

The story

In 2012 a Microsoft employee working on Bing suggested a change to how ad headlines were displayed on the results page. It would take an engineer only a few days to build. Program managers ranked it low among hundreds of ideas, and it sat untouched for more than six months.

Then an engineer noticed how cheap it was to code and simply ran it as an A/B test. Within hours the new version was earning so much more that it set off Bing's "too good to be true" alert, which usually means a bug. It was not a bug. Revenue rose 12% with no harm to the user-experience metrics, worth more than $100 million a year in the US alone. Ron Kohavi and Stefan Thomke later called it the best revenue-generating idea in Bing's history, and the one nobody had rated.

Harvard Business Review, Kohavi and Thomke: The surprising power of online experiments (Sep 2017)
Facebook offices in Menlo Park, California, with the campus buildings and signage
Meta's Menlo Park campus. The retrieval system built here now does much of the audience work that targeting settings used to do.Photo: Anthony Quintano from Honolulu, HI, United States, CC BY 2.0, via Wikimedia Commons

What is the difference between a concept and a tweak?

A concept changes what the ad says or who it speaks to; a tweak changes how it looks while saying the same thing. Concepts are what you test. Tweaks are what you iterate once a concept wins.

LevelWhat changesExample for a ₹1,499 skincare kitTreat as
Angle (concept)The core promise or problem"Fixes tanning from a two-wheeler commute" vs "Dermatologist-made, fragrance-free"New test
Persona (concept)Who the ad speaks toCollege student in Pune vs working mother in KochiNew test
Format (concept)How the story is toldFounder talking to camera vs UGC unboxing vs static before-afterNew test
Proof (concept)Why to believe itLab claim vs 4.6-star review count vs a creator's 30-day diaryNew test
Hook (iteration)First 2 to 3 secondsDifferent opening line, same bodyVariant inside a winning concept
Polish (tweak)Colour, font, music, CTA buttonGreen CTA vs orange CTANot worth a test at this spend

The Bing headline in the story above looks like a tweak, and it made a fortune. The difference is scale: Bing's average experiment had about 20 million users, enough to read a small change. At a Meta budget of a few lakh a month, you cannot read a small change at all, so spend your tests where the differences are large.

How many creatives should you test each week?

Bigger accounts test far more, and they find winners more often. Motion's 2026 Creative Benchmarks report that accounts spending over $1 million a month on Meta launch about 18.8 new creatives a week, against 2.8 for accounts under $10,000, roughly 6.7 times as many. Their hit rate, the share of creatives that become winners, is about 8.8% against 4.0%.

New creatives launched per week and hit rate, by monthly Meta spend
See the data as a table
Creatives launched per weekHit rate (%)
Micro (under $10K)2.84.0
Small ($10K-50K)4.16.4
Medium ($50K-200K)6.68.1
Large ($200K-1M)11.28.6
Enterprise ($1M+)18.88.8

Source: Motion (now Runneth): Creative Benchmarks 2026, scale and volume · Hit rate = share of creatives that become winners, as defined by Motion. Dataset size is not stated on the page.

Read the chart carefully. It is tempting to conclude that volume causes hit rate. It may partly: more launches mean more learning about what works. But big accounts also have better creative teams, more data per ad and stronger brands.

The arithmetic of discovery

What the chart does show is the arithmetic of discovery. At a 6.4% hit rate (Motion's Small tier), you need about 16 launches to expect one winner. At 4 launches a week, that is one winner a month.

Calculated in this post from the 6.4% hit rate and 4.1 launches a week in Motion's Small tier row: 1 / 0.064 is about 16 launches, which is about four weeks at 4 a week.

If your best ad fatigues in three weeks, you are running a deficit.

At about ₹94 to the dollar (Trading Economics, 13 Nov 2026), ₹10 lakh a month is about $10,600, which puts most growing D2C brands in Motion's Small tier. Do not copy an enterprise's volume. Copy its system: a steady weekly cadence, sized to what your budget can actually read.

Why test many ideas cheaply instead of a few ideas thoroughly?

Because in creative, as in product, a small share of ideas creates most of the value, and you cannot predict which ones in advance. The best evidence comes from Microsoft's experimentation platform.

A/B Testing with Fat TailsEduardo M. Azevedo, Alex Deng, José Luis Montiel Olea, Justin Rao, E. Glen Weyl · Journal of Political Economy · 2020

The authors model how a firm should split a fixed pool of experimental traffic across ideas, then estimate the distribution of idea quality from thousands of experiments on Microsoft Bing's EXP platform, where the average experiment had about 20 million users. They find extremely fat tails: the top 2% of ideas account for 74.8% of historical gains. When tails are this fat, a "lean" strategy that tests more ideas with smaller samples beats a few large, high-powered tests, because big outliers show up even in small tests. The limit: these are product and search-ranking changes at Bing, not ads, so the exact numbers will not transfer, but the shape of the argument does.

Read the paper (free)
The top 2% of ideas accounted for 74.8% of historical gains.Azevedo et al., A/B Testing with Fat Tails (2020)

Two lessons follow for Meta creative:

  1. Spend your testing budget on breadth of concepts, not on proving that a 3% difference between two near-identical ads is real.
  2. Shrink small wins. The paper notes that ideas with small measured effects are often lucky draws, while outliers are more likely to be real. A concept that beats your control by 5% on 20 conversions is probably noise. One that halves your cost per purchase is worth a second look.

How much budget does each creative test need?

Each concept needs enough spend to produce roughly 25 or more of the conversion you optimise for, which makes your test budget a function of your cost per conversion.

The reason is simple counting statistics. Conversions behave roughly like a Poisson count, so the relative noise in a count of n is about 1/√n. At 9 conversions that is ±33%, at 25 it is ±20%, and at 100 it is ±10%. Below about 25, you cannot tell a good concept from a lucky one unless the gap is huge.

How sure is a ₹600 CPA read? Noise ≈ 1/√n ₹600 true CPA 9 conversions±33% 25 conversions±20% 100 conversions±10% 400 conversions±5% ₹400₹800 Each 4x more conversions only halves the noise. 25 is where a 1.5x loser stops hiding.
Why the post uses 25 conversions as the reading point. Bars show roughly one standard error either side of a true ₹600 cost per purchase; a real test read can sit anywhere in the bar.

How many concepts can you test each month?

Run your own numbers. A brand spending ₹10 lakh a month with a ₹600 cost per purchase and 20% in testing can read about 13 concepts a month at 25 purchases each. A B2B account with a ₹4,000 cost per qualified lead can read only two or three, which is why lead-gen accounts often test against a cheaper upstream event first and confirm winners on the expensive one later.

A creative testing pipeline Researchreviews, calls, comments Conceptsangle, persona, format Test campaign15-25% of budget Kill rulesat 25+ conversions Scaling campaignwinners graduate here winner Iterate the winnernew hooks, cuts, ratios fatigue signal: frequency up, CTR and CVR down, back to research
A pipeline, not a one-off test. New concepts enter every week; winners graduate and get iterated; fatigue sends the team back to research.

How should the test campaign be structured?

Keep testing separate from scaling, so a new concept is not starved by an old winner that already has months of delivery history. A common structure:

  1. One testing campaign, 15 to 25% of spend. Each new concept goes into its own ad set, or into a fresh ad set of three to five distinct concepts, with the same optimisation event and audience settings as your main campaign. Use broad targeting so the creative does the targeting.
  2. Launch on a fixed day. A weekly batch makes results comparable and keeps the creative team on a rhythm.
  3. Read at a spend threshold, not a date. Judge each concept once it has spent about 25 times your target CPA, or at 7 days, whichever comes later.
  4. Apply kill rules. Pause a concept that has spent 1.5x your target CPA with zero conversions; pause one whose CPA is over 1.5x the control at 25 conversions; mark as a winner one within 0.8x of control CPA at 25+ conversions.
  5. Graduate winners. Duplicate the winning ad, with its post ID so social proof carries over, into the scaling campaign. Leave the test version running or pause it; do not edit it.
  6. Iterate inside the winner. Five new hooks on the winning body, a 4:5 cut and a 9:16 cut, a shorter version. These are variants, not new tests.
A black and white photographic contact sheet with rows of small frames from a 1976 White House photo session
A photographer's contact sheet from 1976. Dozens of frames, one picked. Creative testing is the same idea at the speed of an ad auction.Photo: White House Photographic Office (WHPO) - Schumacher, Public domain, via Wikimedia Commons

Can you trust Meta's reported results for a creative test?

Trust them for ranking concepts against each other inside one campaign; be careful using them as proof that ads caused sales.

The best evidence on why this matters came from Meta's own data.

A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at FacebookBrett Gordon, Florian Zettelmeyer, Neha Bhargava, Dan Chapsky · Kellogg white paper, later in Marketing Science · 2016 (published 2019)

The authors took 12 US advertising lift studies run as randomised controlled trials at Facebook, covering 435 million user-study observations and 1.4 billion impressions, and compared the true lift with what common observational methods would have estimated: exposed versus unexposed comparisons, matching, regression adjustment and before-after tests. The observational methods often missed badly, even with thousands of behavioural variables; in one summary, even a rich model overestimated lift by about 50% on point estimates. The limit: these were 2014-15 campaigns with older delivery systems, but the core problem, that the people who see an ad differ from those who do not, is unchanged.

Read the paper (free)

So use the test campaign to choose concepts, and use proper experiments (Meta's conversion lift, geo holdouts, or a marketing mix model) to decide how much to spend overall. The primer on marketing mix modeling with Meridian and Robyn covers the second part.

Is it creative fatigue, or something else?

Often something else. Before blaming the creative, check:

  • frequency and audience saturation;
  • a change in the optimisation event;
  • a pixel or Conversions API break;
  • seasonality, and auction competition around sales like Diwali and the Big Billion Days.

True fatigue shows a specific pattern: frequency rises among the same people while click-through and conversion rate fall on that ad, and new concepts recover performance. If new concepts fail too, the problem is upstream of creative. The detailed diagnosis is in how often to refresh Meta creative.

How does AI production fit without filling the account with slop?

AI is excellent at the iteration layer and weak at the concept layer. It can produce 20 hooks, five aspect ratios and three voice-overs for a winning concept in an afternoon. It cannot tell you which customer anxiety is worth building a concept around; that comes from reviews, sales calls and comment threads.

The risk is volume without variety. A hundred AI variants of one idea are, to the retrieval system, still one idea, and to the viewer they can look cheap, which costs trust (see what AI slop ads cost a brand).

The rule I work by is that AI should cut repetitive production and raise a small team's effective capacity, and that content volume is not proof of good marketing. At Gnani.ai, AI workflows cut production cycle time by about 60%, and the right use of that time is research and concepts, not more variants.

You can see that work on the work page, and the weekly production system is in an AI ad creative pipeline that ships every week.

Questions people ask

Are creatives burning out faster on Meta now?

Many advertisers report it, but check the cause first. Rising frequency with falling CTR on the same ad is fatigue; falling results across all ads, including new ones, points to tracking, auction or offer problems. A steady pipeline of distinct concepts is the structural answer to genuine fatigue.

Should I test creatives in ABO or CBO?

Either can work. Separate ad sets with fixed budgets (ABO) guarantee each concept gets enough spend to be read; campaign budget optimisation (CBO) spends faster on early favourites and can starve new concepts. Many teams test in ABO or with spend controls and scale winners in CBO or Advantage+.

How many ads should be in one ad set?

Meta no longer recommends a cap of six. What matters is distinct concepts with enough spend each. Three to five genuinely different concepts per test ad set is a practical range for budgets of ₹10 to 50 lakh a month.

How long should a creative test run?

Until each concept has about 25 conversions of your optimisation event, and at least 7 days to cover weekday patterns. Reading earlier leaves roughly ±20% or more noise in the result.

Sources

  1. Meta Engineering: Meta Andromeda, next-gen personalized ads retrieval engine (Dec 2024)
  2. Motion (Runneth): Creative Benchmarks 2026, scale and volume (2026)
  3. Jon Loomer: Meta Andromeda and creative diversification, 7 examples explained (Oct 2025)
  4. Jon Loomer: Meta Andromeda (Aug 2025, updated Dec 2025)
  5. Azevedo, Deng, Montiel Olea, Rao, Weyl: A/B Testing with Fat Tails (2019 version; Journal of Political Economy 2020)
  6. Gordon, Zettelmeyer, Bhargava, Chapsky: A Comparison of Approaches to Advertising Measurement, Kellogg white paper (2016)
  7. Jon Loomer: Choose an audience for each ad creative (Oct 2026)
  8. Harvard Business Review, Ron Kohavi and Stefan Thomke: The surprising power of online experiments (source for the Bing story) (2017)
  9. Jon Loomer: Meta clarifies upcoming placement control changes (Oct 2026)
creative testingMeta adsAndromedaAdvantage+creative diversificationA/B testingperformance marketing
Sapta

Sapta (Saptarshi Sarkar) is a mechanical engineer turned marketer in Bengaluru and a solo touring rider. He writes the seven Saptaverse blogs. Work with him or read more of Marketblog.

Keep reading in Marketblog

From the other blogs