Category Archives: IP addresses

GoogleBot, BingBot – Is That Crawler Real or Fake?

I’m currently running on a shared Dreamhost server. Dreamhost has a monitoring service that  keeps an eye on the total resource usage for each user account. If some user consumes to much resources on the server, the monitoring service starts killing off processes for that user and an email report is sent. This is great since it saves me much of the performance problems caused by other users on the same server.

Some time ago, the resource usage for started hitting the limit but I didn’t notice any additional traffic when I examined the Google Analytics report. This led me to investigate Apache’s access.log file. Here are two example entries from the log: - - [25/Jun/2019:02:37:05 -0700] "GET /library/file/UninstallTP.exe-154295/ HTTP/1.1" 200 17986 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +" - - [25/Jun/2019:02:37:10 -0700] "GET /b/tag/fake-flash-software/ HTTP/1.1" 200 18719 "-" "Barkrowler/0.9 (+"

The first entry ( claims to be the bingbot and the second ( is a crawler called Barkrowler (

When examining the access.log a bunch of questions are raised:

  1. Let’s say the crawler claims to be BingBot or GoogleBot, but is it the real one coming from one of Google’s or Microsoft’s data centers, or is it a bot that falsely set its user agent to GoogleBot or BingBot?
  2. What about all the other bots out there? Their crawling uses quite a lot of resources, but do they bring any value or users to your web site.
  3. What about all the other high usage IP-numbers that claims to be ordinary users? Are their claims correct, or are they just bots in disguise?

How To Determine If a Bot is Fake

Let’s say you see an entry in the log coming from and it claims to be bingbot. How can we determine that the traffic is from a real bingbot? We can do this using the following two steps:

1) First we do a reverse DNS lookup using the IP from the log.

$ host domain name pointer

The DNS responds with [].

2) Then we do a forward DNS lookup on the hostname we got from the reverse lookup.

$ dig +short

So, to summarise: points to [] which is owned by Microsoft. And the [] hostname resolves back to which we started with. Excellent, we now know that we are dealing with a legitimate bingbot.

Another way to check if an IP belongs to bingbot, if you don’t have the host and dig command line tools available, is to use Bing’s Verify Bingbot Tool. You simply type in the IP address, in this case, and solve the captcha.

Verify bingbot tool reports is a real bingbot
Verify bingbot for

I’m not aware of web verification tools for the other search engines such as Google or Yandex. If you know about such a tool, please let me know. – You Got A Visit From GoogleBot

Was troubleshooting some heavy usage on the web site this morning and noticed huge number of connections from in the  Apache HTTP access.log. If you also see connections from in your HTTP log you simply had a visit from the GoogleBot. It uses “Googlebot/2.1” as its User Agent string. Googlebot

If you’d like to get more details about the connections from and have the privilege of shell access to your Apache HTTP log, just run the following command

cat access.log| grep

and you’ll see all connections, the access times and User Agent data, etc.