Tag Archives: MegaIndex.ru

MegaIndex.ru/2.0 megaindex.com/crawler

While examining the access.log at freefixer.com I found around 1000 hits from a bot named MegaIndex.ru operating from the IP address: - - [03/Sep/2019:23:15:50 -0700] "GET /library/file/hkcmd.exe-188/ HTTP/1.1" 200 19377 "-" "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"

Unfortunately, MegaIndex.ru’s crawler page does not clearly explain why it is a good idea to let the bot crawl my site. Nor does it clearly explain what user User-agent name to use in order to explicitly delay or disallow the bot. In addition to this, it sounds as the maximum value for Crawl-delay is 5 seconds.

I hope that the 5 seconds max value is just a typo. I’m going to try to slow down or block the bot with the following entries in robots.txt:

User-agent: MegaIndex.ru
Crawl-delay: 3600


User-agent: MegaIndex.ru
Disallow: /

I did a reverse IP lookup on and the bot is running at clients.your-server.de. reverse ip lookup

I tried a few geolocation services and all report that the server is located in Germany.

Update 24 hours later: When checking the logs again I noticed that MegaCrawler had done more than 6000 requests. That is unacceptable. I’m blocking in the .htaccess file.