188.8.131.52 is requesting content from Freefixer.com approximately every 10 seconds. Quite often, but that’s OK. I would like Freefixer.com to appear at bing.com.
So, how do I know that this in fact is a real Bingbot, and not some unwanted program that scrapes my web site? I’ll use the same procedure as recommended over at Bing Webmasters Tools. That is, a reverse IP lookup on the IP address, and then a forward IP lookup on the results from the reverse lookup. If you end up with the same IP that you started with, and the reverse lookup reports a search.msn.com, you can rest assured that you are dealing with a legitimate bingbot.
If you do an ARIN lookup on 184.108.40.206, you’ll see that Microsoft owns the range starting from 220.127.116.11 to 18.104.22.168. So I assume you can expect bingbots from all the IP addresses.
I’m currently running FreeFixer.com on a shared Dreamhost server. Dreamhost has a monitoring service that keeps an eye on the total resource usage for each user account. If some user consumes to much resources on the server, the monitoring service starts killing off processes for that user and an email report is sent. This is great since it saves me much of the performance problems caused by other users on the same server.
Some time ago, the resource usage for freefixer.com started hitting the limit but I didn’t notice any additional traffic when I examined the Google Analytics report. This led me to investigate Apache’s access.log file. Here are two example entries from the log:
The first entry (22.214.171.124) claims to be the bingbot and the second (126.96.36.199) is a crawler called Barkrowler (exensa.com).
When examining the access.log a bunch of questions are raised:
Let’s say the crawler claims to be BingBot or GoogleBot, but is it the real one coming from one of Google’s or Microsoft’s data centers, or is it a bot that falsely set its user agent to GoogleBot or BingBot?
What about all the other bots out there? Their crawling uses quite a lot of resources, but do they bring any value or users to your web site.
What about all the other high usage IP-numbers that claims to be ordinary users? Are their claims correct, or are they just bots in disguise?
I’ll simply post each IP number that I investigate below and you can check out the details by clicking on it. You can find the list down below.
How To Determine If a Bot is Fake
Let’s say you see an entry in the log coming from 188.8.131.52 and it claims to be bingbot. How can we determine that the traffic is from a real bingbot? We can do this using the following two steps:
1) First we do a reverse DNS lookup using the IP from the log.
$ host 184.108.40.206
252.39.55.157.in-addr.arpa domain name pointer msnbot-157-55-39-252.search.msn.com.
The DNS responds with [msnbot-157-55-39-252.search.msn.com].
2) Then we do a forward DNS lookup on the hostname we got from the reverse lookup.
So, to summarise: 220.127.116.11 points to [msnbot-157-55-39-252.search.msn.com] which is owned by Microsoft. And the [msnbot-157-55-39-252.search.msn.com] hostname resolves back to 18.104.22.168 which we started with. Excellent, we now know that we are dealing with a legitimate bingbot.
Another way to check if an IP belongs to bingbot, if you don’t have the host and dig command line tools available, is to use Bing’s Verify Bingbot Tool. You simply type in the IP address, in this case 22.214.171.124, and solve the captcha.
I’m not aware of web verification tools for the other search engines such as Google or Yandex. If you know about such a tool, please let me know.