Adwords A/B Split Testing Simplified

February 26, 2009


How can I experiment with different ad texts without affecting my ad ranking?

“it will have had time to accrue performance history…” what does that mean?

To make a long story short: >100 impressions is considered enough volume to judge ad performance. If you want the full story, read more...

photo by: Craig Jewell
  • Set your campaign settings to: Rotate
  • Beginning of campaign: test 3 widely varying ads
  • Ongoing: test two ads, my personal preference


By testing only 2 ads, you are able to more quickly arrive at a winner. Remember to consider not only CTR, but more importantly Conversion Rate. If both ads have accrued well over 100 impressions, ie.the control ad has all time: 2,000 impressions versus “new ad” 600 impressions… that should be enough to discount the performance history ranking issue, ok?

  • Delete the “losing ad”
  • Copy/paste “winning ad”
  • Edit wining ad clone… this is now the new ad
  • Repeat and rinse

Support: Hi Dan. thanks for chatting in to Google AdWords. How can I help you today?

dan: yes, re: ads rotation...

Support: Ok.

dan: if i am looking at two ads that have run for 30 days...

dan: I am deleting poor ad and copy paste good ad (control ad) .. and edit...

dan: this is a best practice question

dan: i delete the poor ad... copy/paste the good add and edit

dan: my question is...

Support: Ok. Please go ahead.

dan: first of all, this is an accepted and recommended method of optimization, yes?

dan: i rather just work with 2 ads at a time, not more

Support: Please give me a minute.

dan: sure

Support: Yes. It is if the edit you made relates more closely to your keyword and landing page. This is a recommended method of optimization.

Support: I also suggest you to replace poor performing ads with more relevant ads.

Support: This will for sure improve your account's performance.

dan: ok

dan: my main question is this...

Support: Please go ahead.

dan: the "new ad"... has no history, the "control ad" has history...

dan: can you please explain this problem?

dan: is this a problem?

dan: a colleague has recommended another method...to clone the control twice, pause control and edit one clone...

dan: to create a "true test" due to the history factor?

dan: can you comment on these methods please?

Support: Please give me a minute.

dan: and i have even read about a third strategy...

dan: leaving control active, cloning control 3x and adding one new ad... to give the control more "power"...

Support: Firstly, regarding the 'new ad' and the 'old ad'. The new ad will not have any history as it is treated a s a fresh one after the edits.

Support: I am sorry. I do not understand what you meant by control.

dan: a "true test"

dan: control = old ad

Support: I ll complete answering this question and take up the next.

Support: Ok.

dan: its all the same question

Support: The new ad will not have any history as it is treated a s a fresh one after the edits.

Support: Does this answer the first part of the question?

dan: and it will likely have less quality score and appear lower in rank compared to the control ad?

dan: so this is not a "true test"?

Support: It does not work this way.

dan: what about the new qs update re: ad rank?

dan: do both of these ads, control and new, appear at same position?

Support: How it works is when you make a change, it is treated as a new ad and the quality score history of the old ad is not carried over to the new one.

dan: that is the problem

dan: how will the new ad's rank compare to the control ad?

Support: It has no relation to the old ad as it is treated afresh.

dan: how will the new ad's rank compare to the control ad?

Support: It has no relation to the old ad as it is treated afresh.

dan: how will the new ad's rank compare to the control ad?

Support: I am sorry. I do not understand what you are looking at. What I am trying to convey to is, when you make a change, it has no relation to the old ad and I do not see a reason why it has to be compared to the old ad.

Support: Does this make things clearer?

dan: no

dan: the new ad has no history

dan: will the new ad have a lower qs and thus appear at a lower position compared to the control ad?

dan: i want to create a "true test"...

dan: if the control ad is appearing at position 2 and the new ad is appearing at position 7...

dan: this is not really a fair test, is it?

Support: There is no hard and fast rule that the new ad will appear at a higher or lower position compared to the old ad.

dan: are you following me?

Support: Yes. I am.

Support: Please let me complete.

dan: why do you make that clai?

dan: claim

dan: we both agreed that the new ad has no history...

dan: so why would it be treated the same as the control ad?

dan: can you back up your theory?

Support: When you create a new ad, the relativeness of it to the keywords that it is triggering is calculated and this copy competes with all the other ads in the fray. It is this and nothing else that decides at which position it appears.

Support: Of course it is not treated as the old ad.

Support: That is precisely what I ve been saying throughout the course of this chat.

dan: assuming that the new ad is only slightly different than the control ad...

Support: But still, it is a different ad.

dan: you are saying now that the control ad's history has No Affect on quality its quality score and position in comparison to the new ad, with no hisotry?

dan: for argument sake: let's agree that the "new ad" is only different slightly, ie. singular versus plural word

Support: Yes. The old ads history does not have a relation to the performance of the new ad.

Support: It does not matter how slight the change is.

Support: A change will make it a different ad.

dan: let me ask my question slightly differently...

Support: History does not have an influence on the performance in this case as we are speaking about a different ad altogether.

dan: ok, for arugment sake...

Support: Please go ahead.

dan: lets say the "new ad" is exactly the same as the "control' ad...

dan: the only difference now is that the new ad has no history compared to control ad...

dan: can you answer my question in that case?

dan: for logical argument sake, ok?

Support: Yes. Even if the new ad is the same as the old ad, the fact the you have deleted and created a new one makes it a new one.

Support: This is the way it works.

dan: ok. so how would the "new ad" in this scenario compare in its rank?

dan: to the old ad, which has history

dan: are you telling me that ad history has no affect on ad ranking?

Support: This new ad would again enter the auction afresh. Please note, this new ad may not compete with the same set of ads that the old ad was competing.

dan: why not?

Support: This for sure will affect the ranking of the new ad.

dan: can you explain in more detail how the cloned ad will be treated differently?

Support: Why should it? There are new advertisers everytime. And there are different ads competing for a place on Google's search pages for every individual search query.

dan: than the "control ad"

dan: lets back up one step...

Support: It all boils down to the competition each ad is facing for every new search query.

dan: let's say that my goal is to create a "true test" of 2 rotating ads in an adgroup...

Support: Ok.

dan: how would you suggest i structure my test to accomplish this goal?

Support: What exactly are you referring to by a 'true test.'

dan: comparing apples to apples

Support: Ok.

dan: we agreed that the new ad is treated differently than the control clone...

Support: Let me put it this way, every time there is a search query on Google, there is an auction to decide which ads make the cut.

Support: And as you may be aware of, advertisers keep changing their bids and ads continually.

dan: would you agree that pausing the old and and cloning the old ad and creating a slightly different ad would be a more true test?

Support: Please let me complete.

Support: What this means is, the same ad may not be placed on the same rank for the same keyword query at different points in time.

dan: are you saying it is impossible to complete an a/b test of two ads in an adroup?

Support: Yes. This is because, there are so many changes that advertisers all over the world make to their ads and bids, which will directly impact your ad's ad rank.

dan: yes, but both ads would experience that same effect equally, or not?

Support: These are things that is beyond any one's control.

dan: Wow. It is Impossible to do an A/B split test in Adwords?

Support: You can do it only when you can be rest assured that all advertisers competing for a keyword make no changes whatsoever to their ads and bids. This is a hypothetical situation. So, my answer to your question is no.

dan: but...

dan: both ads in the adroup are affected equally by this uncontrolled circumstances...

dan: correct?

dan: or... does the control ad have a rank advantage over the new ad?

dan: by your reasoning, and quantum physics... experiments are impossible, as the external world is changing

dan: does that make any sense?

Support: Both ads need not be affected equally by this uncontrolled circumstance. This is because, there is no guarantee that the advertisers competing for the same keyword that you are, make the same set of changes and by the same margin to their ads which will impact their ad rank and indirectly yours.

dan: for argument sake...

dan: let's assume that the external is NOT changing, ok?

dan: can we back up and address the question please?

dan: is the new ad's position lower than the control ad? if so, its not a fair test is it?

Support: Yes. If the external environment is not changing, your experiment will work. But, the chances are that, this is hypothetical.

dan: for argument sake, let's agree that both ads are affected equally by the external mileu

dan: is the new ad appearing lower in rank than the control ad, due to "no history"?

dan: if that is true...

Support: That is not true, because, the change in ad rank is only because of the change is external factors.

dan: what about the recent change re: ad rank and qs... you remember the recent update on this?

dan: so you are saying that history has absolutely no affect on the new ad's rank, compared to the control ad?

Support: All I am saying is, it is only the external factors that decide the rank of your new ad in question and not because it has no history.

dan: we agreed, for this hypothetical example, that both ads are experiencing external factors equally

dan: so, in that case... the "new ad" would theoretically appear in same rank as the control ad?

Support: I am not able to give you confirmatory answers for hypothetical situations Dan.

dan: meaning it would be a "true test

dan: ?

Support: Yes.

Support: Is there anything else I can help you with?

dan: would it be possible to verify your position with your technical team, as I plan to make a very controversial blog post on this topic

Support: I am able to confirm everything I said Dan.

dan: the current understanding is that the new ad is at disadvantage

Support: We've covered a lot today, and I hope I've been helpful. If you have further questions, please visit the AdWords Help Center at https://adwords.google.com/support.

Another Chat Conversation...

Support:Thank you for contacting Google AdWords. Please hold a moment while we route your chat to a specialist who will help you with your question: "".

Support: Hi Dan, thanks for chatting in to Google AdWords. How can I help you today?

dan: http://adwords.google.com/support/bin/answer.py?hl=en&answer=87407

Support: Okay.

dan: "if you find that the unedited ad performs better, you can simply delete the new ad. Or, if you find that the new ad performs better, it will have had time to accrue performance history and a higher position by the time you delete the first ad."

dan: i am still unclear, confused

dan: "it will have had time to accrue performance history and a higher position by the time you delete the first ad."

dan: i am still confused about this performance history effect

Support: I am not sure I understand your question.

Support: Please explain this to me again.

dan: i sent you the link

dan: can you tell me in more detail what that means please?

dan: re: performance history

dan: 200 impressions is enough?

dan: 2,000 ?

dan: how can I know

dan: 5,000?

dan: 100?

dan: ii is not clear

dan: at how many impressions is the new add considered to have a similiar rank due to gaining performance history?

dan: my question is re: performance history

dan: of the new ad

Support: There is no exact count of impressions after which an ad is considered to have similar ad rank as the older ad.

dan: ok, but that link included that information as a very important assumption

dan: when deciding which ad performs better

dan: how can the advertiser make a decision on subjective information like this?

dan: the link sent makes an assumption that the advertiser will know when the new ad has gained the appropriate history

Support: I am sure as a seasoned advertiser with AdWords you will be able to decide which ad has performed better over a period of time.

Support: Correct Dan?

dan: no, that is not a correct assumption, not at all

dan: also, i am preparing a blog post on this topic, and I would appreciate more helpful replies

dan: would you like to look at the example we were looking at?

Support: Okay.

dan: maybe you could place yourself in my shoes?

dan: and try to understand my confusion?

Support: I am looking into your campaign now.

dan: two ads

dan: this month

Support: Thank you for holding. I will get back to you with an answer in a moment.

dan: you see the two ads?

Support: I see that you have 2 ad variations in this ad group.

dan: new new ad has less history...

dan: yet is has almost exactly the same ctr...

Support: Both the ads are exactly the same,

Support: Correct?

dan: perhaps this means it is the better ad, because it probably ranked lower?

Support: I mean both the ad texts have the same letters.

dan: not true

dan: the new ad has a different display url

Support: Yes I see that.

Support: I also see that a few words are different.

dan: only the display url is different

dan: the new ad has same ctr...

dan: does this mean it is performing better?

dan: as it was at a disadvantage?

Support: The disadvantage that the second ad has initially because it started late, gets evened out over a period of time.

dan: but i can only "assume" this as i don't really know what "performance history" means?

dan: exactly.. what time, how much time, how many impressions?

dan: how can i make a decision based on subject information like this?

Support: As I mentioned earlier, there is no number of impressions or time after which a new ad is considered at par with the old ad.

Support: May I know what decision would you like to take based on this information?

dan: so you are saying that it is impossible for me to make an objective decision in this case?

dan: sure

dan: which ad is the better performing ad?

dan: a simple question, one that i need to make very often

dan: what if it was slightly less, it might still be the better ad, because it had a disadvantage, yes?

Support: That is a not correct Dan.

dan: perhaps 500 impressions is enough, i don't really know what is considered statistical history

Support: Before I give a complete explanation, may I ask you when did you create these ads?

dan: i cannot say exactly, but I assume the new ad... maybe 7-10 days

Support: Okay.

dan: "have had time to accrue performance history"

dan: what does that mean, objectively

Support: When I said that you have to wait for sometime before judging the new ad it is only to make sure that the sample size (in this case the number clicks for the number of impressions) of the new ad is considerable.

dan: how long must we run ad, or is it based on impression count?

dan: "considerable"

Support: There is nothing like an unfair advantage that a new ad is exposed to.

dan: how can i make a decision based on subjective information

Support: I would say wait till the new ad accrues 100 impressions.

dan: in this case... there is much more than 100, 10x more

dan: are you guessing, as i can guess too?

dan: 100 impressions?

Support: I am not guessing, Dan.

dan: well, if this is true... then my traffic stats would enable me to choose a winner

dan: now you gave me an objective measure... 100 impressions

dan: so this "performance history" is only relevant for very low traffic, not in the example we discussed

dan: in the example we discussed, old ad 3,134 impressions and new ad 1,251 impressions...

dan: this performance history advantage of the old ad is neglible, correct?

Support: Okay.

Support: Lets consider the all time data of impressions and clicks for the two ads.

dan: ok

Support: I see that the old ad has had 5,098 impressions and 145 clicks.

dan: ok

Support: The new ad has 1,255 impressions and 35 clicks.

dan: ok

Support: The CTR for the 1st ad is (145/5098) * 100.

Support: It is 2.84%.

dan: they are almost identical, but conv rate is very different

Support: Similarly the second ads CTR come up to 2.78%.

Support: Conversion depends on factors like landing page.

dan: we can conclude then that old ad is "better"

dan: same landing page

dan: as both ads are well over 100 impressions...

dan: it is a "fair test"?

Support: Yes Dan, the old ad is better.

dan: you have been very helpful

Support: It is indeed a fair test.

dan: can you give me a link supporting your 100 impression claim?

Support: My pleasure, Dan.

Support: Dan, I told you that I was giving you an example and that value 100 need not be the real test.

dan: "example" of what?

Support: For us to divide the two numbers (Clicks and impressions) we need a considerable number for the both the ads in these columns.

Support: Suppose if the second ad gets a click in the first impression itself, it is unfair for us to say that the CTR is '1'.

Support: Correct Dan.

dan: 1 impressions is not significant, agreed

Support: So whenever you think the number of impressions and clicks are significant, you are ready to rate the ads.

Support: I hope this answers your question.

dan: 100 impressions?

dan: are you really sure about that?

dan: i would have guessed 1,000

Support: It's not a harm to wait for 1000 impressions.

Support: However, based on my experience 100 impressions is good enough before we judge an ad and CTR.

Support: I hope that this information was useful to you.

dan: that really helps, thank so much

Support: You are welcome, Dan.

dan: have a great day

Support: You too, Dan.

Add To Google Bookmarks Add To Del.icio.us Add To Reddit Add To Technorati Subscribe to RSS

2 comments:

Anonymous said...

Oh my. Is this a positive AdWords support experience? You know how to make a girl's day....

Dale said...

I didn't read the whole chat - too long. Thanks for the summary at the beginning of the post :)

Post a Comment

Related Posts Plugin for WordPress, Blogger...
x

Get Our Latest Posts Via Email - It's Free

Enter your email address:

Delivered by FeedBurner