Tag Archives: bingbot

207.46.13.179 – bingbot/2.0

Recently I’ve been fixing some performance issues due to the behaviour of some web bots. If you are concerned about 207.46.13.179, I’ll let you right away that this a legitimate Bingbot. Here’s an example from the HTTP access log:

207.46.13.179 - - [04/Sep/2019:21:53:15 -0700] "GET /library/file/mustangser532.exe-206653/ HTTP/1.1" 301 4303 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

207.46.13.179 is requesting content from Freefixer.com approximately every 10 seconds. Quite often, but that’s OK. I would like Freefixer.com to appear at bing.com.

So, how do I know that this in fact is a real Bingbot, and not some unwanted program that scrapes my web site? I’ll use the same procedure as recommended over at Bing Webmasters Tools. That is, a reverse IP lookup on the IP address, and then a forward IP lookup on the results from the reverse lookup. If you end up with the same IP that you started with, and the reverse lookup reports a search.msn.com, you can rest assured that you are dealing with a legitimate bingbot.

207.46.13.179 is the real Bingbot from Microsoft

If you do an ARIN lookup on 207.46.13.179, you’ll see that Microsoft owns the range starting from 207.46.0.0 to 207.46.255.255. So I assume you can expect bingbots from all the IP addresses.

GoogleBot, BingBot – Is That Crawler Real or Fake?

I’m currently running FreeFixer.com on a shared Dreamhost server. Dreamhost has a monitoring service that  keeps an eye on the total resource usage for each user account. If some user consumes to much resources on the server, the monitoring service starts killing off processes for that user and an email report is sent. This is great since it saves me much of the performance problems caused by other users on the same server.

Some time ago, the resource usage for freefixer.com started hitting the limit but I didn’t notice any additional traffic when I examined the Google Analytics report. This led me to investigate Apache’s access.log file. Here are two example entries from the log:

157.55.39.252 - - [25/Jun/2019:02:37:05 -0700] "GET /library/file/UninstallTP.exe-154295/ HTTP/1.1" 200 17986 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
163.172.64.171 - - [25/Jun/2019:02:37:10 -0700] "GET /b/tag/fake-flash-software/ HTTP/1.1" 200 18719 "-" "Barkrowler/0.9 (+http://www.exensa.com/crawl)"

The first entry (157.55.39.252) claims to be the bingbot and the second (163.172.64.171) is a crawler called Barkrowler (exensa.com).

When examining the access.log a bunch of questions are raised:

  1. Let’s say the crawler claims to be BingBot or GoogleBot, but is it the real one coming from one of Google’s or Microsoft’s data centers, or is it a bot that falsely set its user agent to GoogleBot or BingBot?
  2. What about all the other bots out there? Their crawling uses quite a lot of resources, but do they bring any value or users to your web site.
  3. What about all the other high usage IP-numbers that claims to be ordinary users? Are their claims correct, or are they just bots in disguise?

I’ll simply post each IP number that I investigate below and you can check out the details by clicking on it. You can find the list down below.

How To Determine If a Bot is Fake

Let’s say you see an entry in the log coming from 157.55.39.252 and it claims to be bingbot. How can we determine that the traffic is from a real bingbot? We can do this using the following two steps:

1) First we do a reverse DNS lookup using the IP from the log.

$ host 157.55.39.252

252.39.55.157.in-addr.arpa domain name pointer msnbot-157-55-39-252.search.msn.com.

The DNS responds with [msnbot-157-55-39-252.search.msn.com].

2) Then we do a forward DNS lookup on the hostname we got from the reverse lookup.

$ dig +short msnbot-157-55-39-252.search.msn.com

157.55.39.252

So, to summarise: 157.55.39.252 points to [msnbot-157-55-39-252.search.msn.com] which is owned by Microsoft. And the [msnbot-157-55-39-252.search.msn.com] hostname resolves back to 157.55.39.252 which we started with. Excellent, we now know that we are dealing with a legitimate bingbot.

Another way to check if an IP belongs to bingbot, if you don’t have the host and dig command line tools available, is to use Bing’s Verify Bingbot Tool. You simply type in the IP address, in this case 157.55.39.252, and solve the captcha.

Verify bingbot tool reports 157.55.39.252 is a real bingbot
Verify bingbot for 157.55.39.252

I’m not aware of web verification tools for the other search engines such as Google or Yandex. If you know about such a tool, please let me know.

IP Addresses