Let’s say you are doing your first online advertising campaign.  The first terms you advertise in the search engine listing is your organization’s name.  Or the first audience you advertise to on the social networks are the people who are already your fans.

This makes perfect sense.  In the land of concentric circles on a PowerPoint slide, these are the closest in – your favorite people and your most popular search term.  And over time, your wise judgement is validated.  Your brand terms have your highest click-through rates.  And the people who like you the most on social media, shockingly, like you the most in terms of clicking through on your ads.  Yay!  Raises for everyone!

Except… those are the people most likely to have clicked through whether you placed the ad or not.   Are you just getting the people who would have come to your site anyway, except paying for the experience?

eBay wanted to find out (here’s the study we’ll talk about).  For those who don’t remember the Google of  over a decade ago, you could not swing a Beanie Baby or piece of toast with a burn vaguely the shape of the Virgin Mary without seeing an eBay ad for whatever product you sought.

So they stopped advertising on their brand name.  How much traffic do you think they lost from search engines?

Wrong.  Lower.

Turns out only .5% of traffic was lost (.529% to be precise).  The remaining 99.5% of search traffic found that eBay was, not shockingly, still the first result when you searched “eBay” and clicked it anyway.

What about non-brand keywords?  Turns out eBay is the perfect place to test this, because they have a pretty big number of products and, thus, keywords.  They went cold turkey– no ads in 30% of the media markets in the United States.  Attributed sales – sales coming in within 24 hours of clicking on a Google ad link – dropped 72%.  (Why not 100%?  According to the authors: “The remaining ad sales from test DMAs are an artifact of the error both in Google’s ability to determine a user’s location and our determination of the user’s location.”)

OK, that’s attributed sales.  How did total sales go?  The entire regime of paid ads added only .66% to total sales.

As with branded ads, people generally found their way to eBay whether they received advertising or not.  Why?  Even people who bought from eBay 50 times a year or more used paid search ad links to make 4% of their purchases.  In other words, the ads were bringing in people who would have purchased anyway.  When those ads went away, people still found their way there – every 1% drop in search engine ad clicks was associated with a .5% increase in natural search visits and a .23% increase in direct navigation.

So when eBay had been looking at direct attribution, it thought it had 1500% ROI on its ads.  When they actually turned off the ads, they found they had been getting a negative 63% ROI.  Bit of a difference.

Thus,  this week’s theme: are we improving on silence?  So many of the tests we run are this versus that.  We rarely test this versus nothing.  Replace “sales” above with “donations” and “search engine listings” with, well, anything and you could be making the same errors as eBay was (which isn’t bad company to be in, marketing-wise).

Nor do we see the flip side of that coin.  The board that says “cut off mail– we’ll do it all online” soon finds out how many of its online donations were driven or at least prompted by mail.  And vice versa.  DRTV is well-known for not only bringing in direct revenue but having glorious spillover into mail and online channels.  And so on.

The trick is that A versus B will never get you to this type of testing.  Only by being willing to test something versus nothing will tell you if your rising tide lifts all boats or sinks them.  We’ll see into which category social media advertising falls on Wednesday.

Nick

P.S. A sneak preview of Wednesday’s post – we don’t do this type of testing in part because it’s hard.  Facebook, Google, et al make it easy to do A v B because they get paid either way.  If you are interested in a methodology for testing something versus nothing online, check out this analysis of ghost ads.

P.P.S. Would this have been the same case for someone who wasn’t eBay and thus not likely to be listed at or near the top of organic search listings?  Probably not.  Thus, your mileage, as always, will vary even as your need to test it doesn’t vary.