We said photo decks beat video 10×. We were wrong.

For a while we believed, and said out loud, that photo decks outperformed video by roughly ten times. It shaped what we made. It was wrong in two independent ways at once, which is the interesting part.

Error one: the accounts weren't the same

The decks and the videos hadn't run on the same accounts. Different accounts have different audiences, follower counts, warmup histories — and, as we found out separately, wildly different audience geography. Comparing across them measures the accounts at least as much as the format.

The most ordinary confound there is, and we walked into it because the comparison was assembled after the fact from data we already had rather than designed as a test.

Error two: mean versus median

Short-form performance is violently long-tailed. A handful of posts do enormous numbers and the rest cluster low. Take a mean across that and you're mostly reporting whether a breakout happened to land in your sample.

The second number that fell

We'd also published that about 86% of viewers quit on the cover slide of a deck. Also wrong, and it fails for a subtler reason worth understanding, because a lot of people quote numbers like it.

You can estimate a survival curve across deck lengths. What you cannot do is recover the per-slide drop-off rate from that data. Models with very different per-slide behaviour fit the observed numbers identically once you allow for decks differing in quality. The data doesn't distinguish them.

So "86% quit on the cover" was never a measurement. It was one of several hypotheses that fit equally well, reported as though the data had picked it. Answering it properly needs a slide-swap experiment — same deck, same account, cover varied.

What we do now

• Compare formats on the same accounts, or don't compare them.
• Report medians on anything long-tailed. Means describe the outlier.
• Before quoting a rate, ask whether the data could have come out differently under a competing explanation. If not, it isn't evidence.
• Never let a number estimated one way get quoted as though it were measured another way.