How can I check if an IP is a bot or not?
- Thread starter essaid
- Start date
First step would be to check if it's a proxy/datacenter IP or a residential IP.
I can recommend this website and API for that: Proxy & VPN detection API - IPHub.info
Works pretty good in my experience.
I can recommend this website and API for that: Proxy & VPN detection API - IPHub.info
Works pretty good in my experience.
Graybeard
Well-Known Member
Take a list of all the IP's hitting your website and analyze them all.
Or, you can hire 100 monkeys to type them into the form on some website ...
Do you have access to your webserver logs where the traffic is logged? Start there.
How much traffic? a random sample of 2,000 IPs would be enough to make a pretty statistically accurate assessment.
in ssh cut and shuf -n2000 >list
Learn what a reverse lookup is and what it will tell you. hostnames that come back are tell alls
do you understand ssh and server administration at all?
make a bash script to loop and do a lookup on the /24 CIDR of every IP sorted to sort -u (without duplication) |unique -c in a sort will reveal a lot -- many bots will use similar IP C and D blocks (usually belonging to datacenters) 1.2.1-254.1-254 (IPv4)
IPv6 is a real can of worms
Or, you can hire 100 monkeys to type them into the form on some website ...
Do you have access to your webserver logs where the traffic is logged? Start there.
How much traffic? a random sample of 2,000 IPs would be enough to make a pretty statistically accurate assessment.
in ssh cut and shuf -n2000 >list
Learn what a reverse lookup is and what it will tell you. hostnames that come back are tell alls
do you understand ssh and server administration at all?
make a bash script to loop and do a lookup on the /24 CIDR of every IP sorted to sort -u (without duplication) |unique -c in a sort will reveal a lot -- many bots will use similar IP C and D blocks (usually belonging to datacenters) 1.2.1-254.1-254 (IPv4)
IPv6 is a real can of worms
Last edited:
I didn't test it yet, but looks good, thank you
Thank you Graybeard.
Yes, I have a root access to my server and have all server logs.
Also I know about ssh cmd little bit, I'm not good on networkside, but I got what you said.
Good idea, thank you again
Hmm. You need to check the IP address against known proxy,bot database. IP2Location
Also you would want to check the HTTP Request Header and check the User-Agent. that would give some details about what made the click (Web browser, Search Engine Spider..etc)
Also you would want to check the HTTP Request Header and check the User-Agent. that would give some details about what made the click (Web browser, Search Engine Spider..etc)
Graybeard
Well-Known Member
It is so damn easy to forge the user-agent LMAO.
From my experience with PCI-DSS billing servers these commercial data bases are not that effective -- plenty slip through the cracks trying to, and sometimes succeeding, with fraudulent payment transactions. Data bases we paid near $400/mo for.
I penetrated near 100% of the websites I connected with >>>Ask Me Anything - Private proxy-cloaker FREE! (<LOL>) Now, I made no attempts to hack or defraud some way -- but give me your ad links and I'll try and we can see if it's counted
Dare Ya >:=D
From my experience with PCI-DSS billing servers these commercial data bases are not that effective -- plenty slip through the cracks trying to, and sometimes succeeding, with fraudulent payment transactions. Data bases we paid near $400/mo for.
I penetrated near 100% of the websites I connected with >>>Ask Me Anything - Private proxy-cloaker FREE! (<LOL>) Now, I made no attempts to hack or defraud some way -- but give me your ad links and I'll try and we can see if it's counted
Dare Ya >:=DWhat do bots try to do and gain by coming to websites with affiliate marketing?
Graybeard
Well-Known Member
I has occasion to match the MaxMind Geo-City-Light (free distrubution) data base against my own proprietary ban list by AS networks assigned to known data centers (bots and servers). My list is by no means exhaustive however -- of 140 million CIDR assignments in the MaxMind Geo-City-Light my lists found 40 million banned CIDRs.
The remaining 100 Million were thought to be normal user IPs (residential IPs)
I spot checked manually and some were hereto unknown possible 'bougies' (suspects).
What does this all mean you ask? I means maybe 1/3 of the IP distribution in the US is to be suspect.
I can just white list my own server's IPs and the IP of the services I want to do business with or communicate with.
The remaining 100 Million were thought to be normal user IPs (residential IPs)
I spot checked manually and some were hereto unknown possible 'bougies' (suspects).
What does this all mean you ask? I means maybe 1/3 of the IP distribution in the US is to be suspect.
I can just white list my own server's IPs and the IP of the services I want to do business with or communicate with.
Graybeard
Well-Known Member
- SEO scraping bots: copy your webpages and check your links (for your competitor)
- Blackhat scraping bots: stealing your content, copy and code
- The same type bots may follow the ads that you buy
- Click bots that consume your CPC/PPC budget
Up to 40% is the estimate of all of this type of traffic on the internet -- no benefit to the webmaster community -- if you have any website you belong to the 'community
Similar threads
