Monthly Archives: September 2019

GoogleBot, BingBot – Is That Crawler Real or Fake?

I’m currently running on a shared Dreamhost server. Dreamhost has a monitoring service that  keeps an eye on the total resource usage for each user account. If some user consumes to much resources on the server, the monitoring service starts killing off processes for that user and an email report is sent. This is great since it saves me much of the performance problems caused by other users on the same server.

Some time ago, the resource usage for started hitting the limit but I didn’t notice any additional traffic when I examined the Google Analytics report. This led me to investigate Apache’s access.log file. Here are two example entries from the log: - - [25/Jun/2019:02:37:05 -0700] "GET /library/file/UninstallTP.exe-154295/ HTTP/1.1" 200 17986 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +" - - [25/Jun/2019:02:37:10 -0700] "GET /b/tag/fake-flash-software/ HTTP/1.1" 200 18719 "-" "Barkrowler/0.9 (+"

The first entry ( claims to be the bingbot and the second ( is a crawler called Barkrowler (

When examining the access.log a bunch of questions are raised:

  1. Let’s say the crawler claims to be BingBot or GoogleBot, but is it the real one coming from one of Google’s or Microsoft’s data centers, or is it a bot that falsely set its user agent to GoogleBot or BingBot?
  2. What about all the other bots out there? Their crawling uses quite a lot of resources, but do they bring any value or users to your web site.
  3. What about all the other high usage IP-numbers that claims to be ordinary users? Are their claims correct, or are they just bots in disguise?

How To Determine If a Bot is Fake

Let’s say you see an entry in the log coming from and it claims to be bingbot. How can we determine that the traffic is from a real bingbot? We can do this using the following two steps:

1) First we do a reverse DNS lookup using the IP from the log.

$ host domain name pointer

The DNS responds with [].

2) Then we do a forward DNS lookup on the hostname we got from the reverse lookup.

$ dig +short

So, to summarise: points to [] which is owned by Microsoft. And the [] hostname resolves back to which we started with. Excellent, we now know that we are dealing with a legitimate bingbot.

Another way to check if an IP belongs to bingbot, if you don’t have the host and dig command line tools available, is to use Bing’s Verify Bingbot Tool. You simply type in the IP address, in this case, and solve the captcha.

Verify bingbot tool reports is a real bingbot
Verify bingbot for

I’m not aware of web verification tools for the other search engines such as Google or Yandex. If you know about such a tool, please let me know.