# Similarity is not prediction

> We backtested whether creative embeddings can forecast ad performance. Once brand and objective base rates were controlled for, the signal collapsed. Here is what we do with that.
- **Author**: Apurv
- **Published**: 2026-09-05
- **Category**: Creative strategy
- **URL**: https://marxx.ai/posts/similarity-is-not-prediction

---

Every creative intelligence tool is under pressure to say the same sentence:
our AI knows which ad is going to win. It is a good sentence. It is also, as
far as we can tell from our own data, not true.

So we tested it properly before we put it on a website.

## What we ran

We took creative embeddings across a large pool of ads and asked a simple
question: given only the creative, can we predict how it will perform?

The first result looked spectacular. Pooled across the whole set, a
classifier separating high-CTR from low-CTR creative hit an AUC of 0.94. If
you stop there, you have a product claim.

We did not stop there.

## Where it went

The pool contained ads with different objectives, from different brands, at
different spend levels. Traffic campaigns have structurally different
click-through rates than conversion campaigns. Some brands simply get more
clicks than others, regardless of what the creative does.

An embedding encodes all of that. It knows a traffic ad looks like a traffic
ad. So the classifier was not learning "this creative is persuasive". It was
learning "this is a traffic ad from a brand with a high baseline", and the
CTR label was largely a restatement of the same fact.

When we demeaned by brand and objective, so that the model had to explain
variance *within* a brand and *within* an objective, AUC fell to **0.36**.
Below chance. The signal was base rates wearing a costume.

## What embeddings are actually good for

This is not a case against embeddings. It is a case against one specific claim.

Embeddings are excellent at **similarity**. Which of our ads are close to this
one. Which competitor creative occupies the same space as ours. What has this
account never made anything near. Those are all questions about geometry, and
geometry is exactly what a vector space gives you.

They are poor at **forecasting**, at least at the volumes most advertisers
operate at, because performance is dominated by factors the creative does not
contain: audience, budget, bid, seasonality, offer, landing page, and the
platform's own delivery decisions.

## What we say instead

Marxx tells you what ran, what is decaying, what your competitors are doing,
and what your account has never tested. Diagnosis and coverage. Every one of
those is verifiable against the data the same day.

We do not tell you which ad will win, because when we checked, we could not.

If a vendor tells you otherwise, ask them for the demeaned number.

---
- [More Creative strategy articles](https://marxx.ai/posts/category/creative-strategy)
- [All articles](https://marxx.ai/posts)