A-B Test Very Close Results?

[Anthony.]

Hey guys!

I've been running an A/B test between 2 variables for 5 days now, one day A is better and one day B is better, but over the time they are both near enough the same, EPV is identical almost.

Do I keep it running for more data till significance?
A: -53% ROI
B: -54% ROI

Today(still running though):
A: -72% ROI
B: -30% ROI

Yesterday:
A: -61% ROI
B: -69% ROI

22nd Feb:
A: -7% ROI
B: -61% ROI

21st Feb:
A: -69% ROI
B: -53% ROI

Each day is clearly a rollercoaster, but I've let them compete with eachother but looking at the data since starting this test (there were 3 but one performed terribly) these 2 are very close and not significant.

What next? Leave it alone till one wins, or start looking at other metrics to decide which looks the best.

Advice appreciated

 

[Graybeard]

make 'B' a more radical change.
If you are just doing incremental changes you may be getting a *see-saw*

Like with or without video or images or?

 

[Anthony.]

make 'B' a more radical change.
If you are just doing incremental changes you may be getting a *see-saw*

Like with or without video or images or?

Thanks for the reply!

It's 2 different angles, approaching the niche in different ways - one performed terrible (I was testing 3) But these two are totally different, yet very close with stats.

Now so far today:
A: -39% ROI
B: -31% ROI

Hmm, Maybe just leave them running until it's 7 days? That way I have then ran both during the whole week - seeing different performances on different days.
Just not sure it will hit significance.
After this test, whichever wins I will be opening up that angle into more ways of coming across to the visitor. So choosing the correct winner is crucial to the rest of the campain

 

[Anthony.]

What is the true audience overlap?
You are choosing 1 success out of 2 successes ... what might be lost? Why not just keep both as 'co-winners'?
That is another way to look at this ...

Hmm good point, I just wanted to get to choose the significant best and expand on it keeping everything tidy.
Taking the campaign through different phases and scaling.

 

[Graybeard]

What is the true audience overlap?
You are choosing 1 success out of 2 successes ... what might be lost? Why not just keep both as 'co-winners'?
That is another way to look at this ...

 

[Anthony.]

• Run with 2 winners and the compare each winner alone just look at the CR for each and just who is the PERSON converted;
• is there some overlap in attributes? if so what is the %?
• there is no rule about only having just 1 winner in a test.

Green people prefer A and Purple people prefer B
Both Green and Purple people have MONEY.
By just scaling one page how many Greens or Purples will you lose? Is that going to be a net gain when scaling just one landing page?

Don't leave any substantial money on the table.
Or, to quote the age-old adage: "Don't throw the baby out with the bathwater"

Why can't you scale 2 landing pages -- if you can optimize the audience maybe it is more easily done?

Very well said, yeah I understand and it doesn't really bother me about scaling two, if anything one may help if another performs less on daily basis.
I guess I'll just expand both and see how more ideas for each angle affect the performance.

 

[Graybeard]

• Run with 2 winners and the compare each winner alone just look at the CR for each and just who is the PERSON converted;
• is there some overlap in attributes? if so what is the %?
• there is no rule about only having just 1 winner in a test.

Green people prefer A and Purple people prefer B
Both Green and Purple people have MONEY.
By just scaling one page how many Greens or Purples will you lose? Is that going to be a net gain when scaling just one landing page?

Don't leave any substantial money on the table.
Or, to quote the age-old adage: "Don't throw the baby out with the bathwater"

Why can't you scale 2 landing pages -- if you can optimize the audience maybe it is more easily done?