Bollywood trailers and A/B tests are the same experiment in spirit: show audiences a version, measure what they do, and push the version that wins. They differ in one ingredient, randomisation. An A/B test shows two versions to random halves of an audience at the same moment, so the difference is caused by the version; a teaser-then-trailer campaign shows versions one after another, so the calendar, the star's birthday and the release date are all mixed into the result. Film marketing became a true A/B test only when platforms began splitting the audience for it.
- 13 points: the jump in Animal's Ormax "Appeal" score in its trailer week (2023).
- ₹25.9 → ₹48.7 crore: Ormax's first-day forecast before and after that trailer.
- n = (4rσ/Δ)²: Kohavi's rule for how many viewers a fair test needs.
- With these example inputs, about 3.5 lakh impressions detect a 10% lift between three trailer thumbnails at a 4% click rate.
- 82% of Netflix browsers' focus went to artwork, so it A/B tests the images.
This is a collision post: two worlds that look unrelated, the Friday-release business of Hindi cinema and the button-colour tests of tech companies, solved with the same statistics. The hard numbers come from Ormax Media, Microsoft's experimentation team, Netflix and a 2023 study of 413 YouTube trailers.
How are Bollywood trailers and A/B tests the same experiment?
Both expose an audience to a variant, measure intent or action, and decide what to ship next. A campaign of teaser, trailer, songs and promos is a sequence of variants, read through tracking surveys and view counts.
Animal released its teaser on 28 September 2023 and its trailer on 23 November, eight days before the 1 December release. Ormax Media tracked the film on its Cinematix tool, which measures "Appeal", the share of aware audiences who definitely intend to watch in a theatre. Awareness was already 90%. Across the trailer week, Appeal rose by 13 points, a 22% jump, and Ormax's first-day forecast climbed from ₹25.9 crore to ₹48.7 crore. The film then opened above ₹50 crore nett for the Hindi version on day one. Ormax's reading: the trailer brought in new audiences who had not been interested before.
Ormax Media: Case study, box office impact of an effective trailer (Dec 2023)That is a measured treatment, a measured response and a decision. What it lacks is a control group: nobody was randomly kept from seeing the trailer in the same week.
| Element | Film campaign | A/B test |
|---|---|---|
| Variants | Teaser, trailer 1, trailer 2, songs, promos | Version A and version B |
| Who sees what | Everyone, in sequence | Random split, at the same time |
| Outcome measured | Views, tracking surveys (awareness, appeal), first-day forecast | One overall evaluation criterion: clicks, watch time, sales |
| Decision | Which assets to push, how to cut the next one, release strategy | Ship the winner |
| Biggest weakness | Time and hype are confounded with the asset | Needs enough sample to detect small effects |
What does an A/B test need that a trailer launch doesn't have?
Two things: random assignment and enough people in each arm. Microsoft's experimentation team gave a simple rule for the second.
Ron Kohavi, Randal Henne and Dan Sommerfield published a practical guide in 2007 with a rule of thumb for 95% confidence and 90% power. They titled it after the HiPPO, the "highest paid person's opinion", which experiments are meant to overrule.
Assumptions and variables
- \( n \): total viewers needed across all variants.
- \( r \): number of variants, roughly equal in size.
- \( p \): baseline click-through rate on a trailer thumbnail. I assume 4% for the worked example; your channel's number will differ.
- \( \sigma \): standard deviation of a click, a Bernoulli event, so \( \sigma = \sqrt{p(1-p)} \).
- \( \Delta \): the smallest difference worth detecting. I use a 10% relative lift, \( \Delta = 0.1p \).
The worked derivation
- Spread of one click. \( \sigma = \sqrt{0.04 \times 0.96} = 0.196 \).
- Smallest lift. \( \Delta = 0.10 \times 0.04 = 0.004 \), from 4.0% to 4.4%.
Calculated in this post with these example inputs (a 4% click rate and a 10% relative lift), using the Kohavi et al. (2007) sample-size rule.
- Two cuts. \( n = (4 \times 2 \times 0.196 / 0.004)^2 = 392^2 \approx 1.54 \) lakh impressions.
- Three cuts. \( n = (4 \times 3 \times 0.196 / 0.004)^2 = 588^2 \approx 3.46 \) lakh impressions.
- Check against the source. Kohavi's own example, a 5% purchase rate and a 5% change, needs under 5 lakh users; relaxing to a 20% change cuts that 16-fold to 30,400.
The sensitivity lives in the square. Halve the lift you want to detect, 5% instead of 10%, and three cuts need about 13.8 lakh impressions, four times as many. Raise the baseline click rate and the requirement falls, because rarer events are noisier.
Calculated in this post with the same example inputs (a 4% click rate, three cuts) and the Kohavi et al. (2007) rule, changing only the lift from 10% to 5%.
How many viewers does your trailer test need?
Use this to check what you just read. At my assumed ₹100 CPM, a fair three-cut test costs about ₹35,000 of media, small change next to a film's marketing budget.
A Microsoft team's guide to running online A/B tests: choosing one overall evaluation criterion, randomising by user with sound hashing, and sizing samples. It gives the rule \( n = (4r\sigma/\Delta)^2 \) for 95% confidence and 90% power, with worked e-commerce examples (1.6 million users to detect a 5% revenue change, under 5 lakh for a 5% conversion change). The limit: the rule slightly overestimates for large samples, and web experiments assume instant, measurable outcomes, unlike a film releasing weeks later.
Read the paper (free)Why is a trailer campaign a weaker experiment than an A/B test?
Because the trailer and the calendar arrive together. Without a control group, you cannot separate what the trailer did from what the approaching release would have done anyway.
Econometricians handle this with difference-in-differences: compare the change in a treated group with the change in an untreated one over the same weeks.
- What Ormax saw. Appeal rose 13 points, a 22% rise, so the base was about \( 13 / 0.22 \approx 59 \) and the new level about 72 (my arithmetic from Ormax's figures).
- Missing term. The control change \( Y^{C}_{post} - Y^{C}_{pre} \) is the drift that release week alone would have produced. Nobody measured it.
- Sensitivity. If that drift were 0 points, the trailer added all 13. If it were 5, the trailer added 8. If it were 13, the trailer added nothing and simply rode the calendar.
Ormax's own reading, that the trailer pulled in new audiences, is plausible because the forecast nearly doubled in a week. But the structure of the data cannot prove it. That is the same trap as reading a sales jump after a TV burst, the problem marketing mix modelling and incrementality tests exist to solve.
What did film marketing figure out early?
Film marketers learned before tech did that the packaging decides whether anyone samples the product, and that it can be tested before launch.
Pre-release testing in India
Ormax Media runs content testing for films and shows, inviting viewers from a large online panel, to get an early estimate of the Ormax Power Rating a title will earn after release. It deliberately tests trailers and other marketing assets separately, with a tool called Campaign Express, because loading film research with asset questions tires respondents. That is good experimental hygiene: one question per test.
Streaming turned it into a true A/B test
Netflix's product creative team reported in 2016 that artwork took over 82% of members' focus while browsing, that members spent about 1.8 seconds considering each title, and that a member not engaged within 90 seconds was likely to leave. It ran randomised tests on images: faces showing complex emotion beat stoic ones, recognisable villains could win, and groups of more than three people tended to lose. It also found that the winning image differed by country, for example for Sense8, which is the argument for choosing artwork per audience rather than one poster for all.
The bandit: testing and earning at once
A multi-armed bandit is a test that shifts traffic toward the winner while it is still learning, trading exploration against exploitation. Thompson sampling, the common method, shows each version with the probability that it is the best given the data so far:
A film campaign already behaves like a slow, human bandit: the song that trends gets more ad money, the dull promo is quietly dropped. The difference is that a platform bandit decides every few seconds on randomised traffic; a producer decides every few days on mixed signals.

What makes a trailer campaign work, by the data?
A 2023 study of YouTube trailers found that how consistently people talked about a film across its trailers predicted opening weekend better than how much they talked.
They collected about 1.3 million YouTube comments on 413 trailers for 143 wide-release films (2015 to 2017), measured how uniform the language was across a film's trailers with tf-idf, and regressed log opening-weekend box office on it. Consistency had a standardised beta of 0.251, second only to screen count (0.414), and comment volume lost significance once it was added; adjusted R² rose from 0.603 to 0.649. An online experiment with 402 participants supported the mechanism. The limit: Hollywood blockbusters before streaming, and no hold-out test of prediction.
Read the paper (free)See the data as a table
| Standardised beta | |
|---|---|
| Screen count | 0.414 |
| Consistency of trailer comments | 0.251 |
| Volume of trailer comments | 0.031 |
Source: Kampani and Nicolaides, Frontiers in Communication (2023) · Valence, number of trailers and star presence were not significant.
Read as an A/B lesson, this says the winning variant is not the loudest one but the one that keeps telling the same story. A campaign of three trailers that each pitch a different film confuses the audience; three cuts of one clear promise compound. The same logic drives creative testing on Meta: test executions of one idea, not three unrelated ideas.
The winning variant is not the loudest one, but the one that keeps telling the same story.Reading Kampani and Nicolaides (2023) as an A/B result
Can you A/B test the film itself?
You can now, and it has already started a fight. In July 2025 Eros said it would re-release the 2013 film Raanjhanaa on 1 August with an AI-generated alternate ending in which the story ends happily. Director Aanand L Rai said the change was made without his consent; Eros argued that as copyright owner it was entitled to a clearly labelled reinterpretation.
In experimental terms, that is a variant B of the product, not of the ad. Here is what changes when the treatment is the work itself:
- Consent: creators' moral claims sit on top of the copyright owner's legal ones.
- Contamination: audiences talk, so arms of the test leak into each other through word of mouth.
- Measurement: a re-release has no clean baseline; the original had a different year, price and screen count.
So, are trailers and A/B tests the same experiment?
The logic is identical: variants, a measured response, a decision. The design is not. A teaser-to-release sequence is an uncontrolled before-and-after study, so Animal's 13-point trailer-week jump cannot be split between the trailer and the calendar. Add a random split, which YouTube's Test & Compare and paid-media holdouts now allow, and a trailer test becomes a true A/B test: with these example inputs about 3.5 lakh impressions settle a 10% difference between three thumbnails at a 4% click rate, roughly ₹35,000 of media at an assumed ₹100 CPM.
My suggestion for any launch, film or product: cut two versions of the hero asset, randomise them for the first 48 hours, then pour money behind the winner. The film business has been doing the second half of that for decades; the first half is now cheap. For more on what Indian audiences actually watch, see the most-viewed Indian ads on YouTube, and for how this kind of AI-built workflow is set up, applied AI.
What's new
- India's box office crosses ₹13,395 crore in 2025; Hindi films hit a record ₹5,504 crore Ormax Media, Jan 2026
- YouTube begins letting some creators A/B test titles as well as thumbnails Tubefilter, Jul 2025
- Eros defends AI-altered ending for the Raanjhanaa re-release after director objects The Week, Jul 2025
What if next?
- A wedding card is a perfect piece of direct mail Named, hand-delivered and dated: what email forgot.
- Why a street vendor's shout beats your headline The inverse-square law gives a hawker about 6 metres to make the sale.
- A peacock's tail is an ad budget Why a big launch campaign is believed, and when it stops being believed.
Questions people ask
Do trailers actually increase box office?
The evidence is correlational. Ormax tracked a 13-point rise in Animal's Appeal in its trailer week and a forecast jump from ₹25.9 to ₹48.7 crore; a 2023 study found that consistent audience response across trailers predicted opening weekend. Neither isolates the trailer from the release calendar, which only a randomised holdout can do.
How many people do you need for an A/B test?
Kohavi's rule is \( n = (4r\sigma/\Delta)^2 \) for 95% confidence and 90% power. With these example inputs (three variants, a 4% click rate and a 10% relative lift), that is about 3.5 lakh impressions in total.
What is the difference between an A/B test and a multi-armed bandit?
An A/B test splits traffic evenly until the end, then picks a winner. A bandit shifts traffic toward the better version while it learns, earning more during the test but giving a less precise final estimate.
Can YouTube A/B test thumbnails and titles?
Yes. YouTube's Test & Compare feature, announced in 2023 and rolled out from 2024, tests thumbnails; in July 2025 YouTube began experimenting with title tests for a small share of creators.
Sources
- Ormax Media: Case study, box office impact of an effective trailer (Dec 2023; source for the 13-point Appeal rise, 90% awareness and ₹25.9 to ₹48.7 crore forecast)
- Wikipedia: Animal (2023 Indian film) (source for the teaser, trailer and release dates)
- Ormax Media: The Ormax Box Office Report 2025 (Jan 2026; source for ₹13,395 crore, ₹5,504 crore, 83.2 crore footfalls and ₹161 ATP)
- Kohavi, Henne and Sommerfield, KDD 2007: Practical Guide to Controlled Experiments on the Web (source for the sample-size rule and worked examples)
- Kampani and Nicolaides, Frontiers in Communication: Information consistency as response to pre-launch advertising communications (2023; source for the trailer-comment regression)
- Netflix: The power of a picture (May 2016; source for 82% focus, 1.8 seconds and artwork test findings)
- Ormax Media: Content testing, frequently asked questions (source for testing trailers separately with Campaign Express)
- Ormax Media: Content testing FAQ (source for the online panel and early OPR estimate)
- Wikipedia: Multi-armed bandit (source for bandits and Thompson sampling)
- Wikipedia: Difference in differences (source for the estimator)
- The Week: Eros responds to Aanand L Rai over the Raanjhanaa AI ending (Jul 2025)
- Tubefilter: YouTube expands Test & Compare to titles (Jul 2025; source for the 2023 announcement and 2024 rollout)
