How does ad spy software work

[J. Miguel]

Hi,
My very first post here, other than the intro.

As soon as I got to affiliate fix forum, I started to see a lot of ads and talks about ad spy tools.... as a dev, I'm just curious to know how it works under the woods.

I bet ad networks do not provide this info through their APIs, or do they? So I guess these tools are based on scrapping the web for the Ads?

If anyone has a general idea, could you share?

I ask this because I think is a great idea to spy on ads, but since I'm a developer, I usually go for creating my own tools instead of buying them, as long as there's no much effort on doing that. So, I'm asking if anyone has a general idea of how these Spy tools work just to assess if I buy one or build my own for my own consumption.

 

[Graybeard]

Sure it does, people that need ideas buy their SaaS platform

The best way to find out is to do a few free trials I guess (if you can find any).

The assumption they seem to be working on is if some ad is running long term it is a ''evergreen'' product?
Seems to me that a product that is ''trending'' has better possibilities and less competition.

• What I wonder is just how to they get their data?
  
• Are ad networks selling your data to your competitors.

Q. How would you really know what an ad makes/converts?
A. You cannot.​

 

[J. Miguel]

Thanks. It seems that is they scrapping webpages and look at the ads that are being shown?

Don't know if publishers are selling the data, at leat not to some of the tool owners. That data can cost millions and a lot of tools were made in a budget. I'm just guessing.... but would be nice to know how it works, what's the source of the data...

 

[J. Miguel]

I did some digging and have a theory of using bots to crawl websites.

Let's say for example that I want to know about ads for "learn guitar online lessons"

I would run multiple bots with different IPs, countries, and user-agents (Chrome, IE, Android, IOs, etc) and crawl a series of websites, let's say every hour or so. These websites are a predefined database of Website URL and Categories, for example:

ultimate-guitar . com | music;guitar;

I need a database of publishers that publish on that website, and I can build it from crawling and a bit of manual work to clean up. Then I would look for ads from those networks. The links will be enough to know the provider, for example, if it is https: // ads.google.com then I know that the provider is AdWords.

I would click in that link to see where it follows and gather some more info about the landing page and add it to the db

Regarding the "popularity" of the ad, I would take metrics about the frequency it shows on the websites and placement, for example.

Meanwhile, I was able to build a DB with ads related with "guitar" generally because the I have no idea to understand if the AD was related with "learn guitar lessons online", but... I would then analyze the landing pages, apply some AI and get to the niche topic, with fewer errors.


Maybe is how it works, roughly?

 

[TheExplorer]

I did some digging and have a theory of using bots to crawl websites.... Maybe is how it works, roughly?

You are almost there... plus I would also add browser emulation behavior to my crawlers.

But while it looks very easy at least in the theory, I think it would take lots of resources and money to build something which can be really sold as SAS.

 

[Graybeard]

• What IPs?
  
• Data Center IPs are easy to ban.
• What ads? -- most network ads are injected remotely with a JavaScript
  
• What bot? PhantomJS, Puppeteer?

 

[J. Miguel]

• What IPs?
  
• Data Center IPs are easy to ban.
• What ads? -- most network ads are injected remotely with a JavaScript
  
• What bot? PhantomJS, Puppeteer?

What IPs?
I could install malware on peoples' computers. LOL, joking. Am I? Whenever they browse the internet, regardless of the site, my malware would do the scrapping and send results back to my servers. Or, ideally, pay people to install my software in their browsers... there are these ads everywhere "get paid just for browsing the internet". Also, having IPs from different parts of the world it what makes possible to gather data based on the country, ISP, Mobile Operator, etc...

What ads? -- most network ads are injected remotely with a JavaScript
Everything goes to the DOM, otherwise, browsers do not show the ad contents

What bot? PhantomJS, Puppeteer?
None of those, of course. I said I would build the bots...

 

[J. Miguel]

You are almost there... plus I would also add browser emulation behavior to my crawlers.

But while it looks very easy at least in the theory, I think it would take lots of resources and money to build something which can be really sold as SAS.

Not "lots" of money as building the infrastructure for Uber, for example. Is not rocket science, but yeah not a one week task.

 

[Aliza johnz]

When it comes to competition, the first thing any business wants to know is if their competitor is using something and how well are they using it.

 

[J. Miguel]

When it comes to competition, the first thing any business wants to know is if their competitor is using something and how well are they using it.

That's a fact! How the data is gathered was the question.

 

[J. Miguel]

Interesting is also that these bots frequently click the ads that one publishes in the "spied" ad networks and advertisers end up paying to have these bots doing business.

 

[J. Miguel]

I know of one Free Proxy that it's users allow their idle time, while on wireless or Ethernet, to allow backdoor requests from paid commercial users. That only service costs $500/mo+ depending on the usage -- they have 4 million residential IPs.

• As far as malware goes that is a plan for failure -- in some places that is a civil tort or even criminal.

Reinventing the wheel and then selling into a market where you have no reputation is going an uphill marketing battle. In the digital world; when people can't (or are too lazy) to create they try to copy -- a few succeed but most fail -- they have nothing new to offer.

If you are as good as you say (or think you are) why not just create a SaaS with user configured bots and sell that?

Did I tickle some part of your brain where you don't like it to be tickled? My initial question was out of curiosity. I have no interest whatsoever to do a SaaS on this, especially when the business is a bit shady. I started the thread with:

"So, I'm asking if anyone has a general idea of how these Spy tools work just to assess if I buy one or build my own for my own consumption."

Was trying to understand how it works, just that.

But is interesting your statement: "In the digital world; when people can't (or are too lazy) to create they try to copy -- a few succeed but most fail -- they have nothing new to offer." So, what's those spy tools for?? Oh, wait to copy what others have done?

 

[J. Miguel]

and in how the notes are played (clean notes vs noisy notes), I know because I play the guitar, with a lot of noisy notes ahah

 

[Graybeard]

I know of one Free Proxy that it's users allow their idle time, while on wireless or Ethernet, to allow backdoor requests from paid commercial users. That only service costs $500/mo+ depending on the usage -- they have 4 million residential IPs.

• As far as malware goes that is a plan for failure -- in some places that is a civil tort or even criminal.

Reinventing the wheel and then selling into a market where you have no reputation is going an uphill marketing battle. In the digital world; when people can't (or are too lazy) to create they try to copy -- a few succeed but most fail -- they have nothing new to offer.

If you are as good as you say (or think you are) why not just create a SaaS with user configured bots and sell that?

 

[Graybeard]

There is no note that has never been played -- the secret is in the composition and the marketing.

All that I have seen is a lot of pictures (screencaps) of ads along with what is imagined as big data that people complain is out of date.