While examining the access.log at freefixer.com I found around 1000 hits from a bot named MegaIndex.ru operating from the 18.104.22.168 IP address:
22.214.171.124 - - [03/Sep/2019:23:15:50 -0700] "GET /library/file/hkcmd.exe-188/ HTTP/1.1" 200 19377 "-" "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"
Unfortunately, MegaIndex.ru’s crawler page does not clearly explain why it is a good idea to let the bot crawl my site. Nor does it clearly explain what user User-agent name to use in order to explicitly delay or disallow the bot. In addition to this, it sounds as the maximum value for Crawl-delay is 5 seconds.
I hope that the 5 seconds max value is just a typo. I’m going to try to slow down or block the bot with the following entries in robots.txt:
User-agent: MegaIndex.ru Crawl-delay: 3600
User-agent: MegaIndex.ru Disallow: /
I did a reverse IP lookup on 126.96.36.199 and the bot is running at clients.your-server.de.
I tried a few geolocation services and all report that the 188.8.131.52 server is located in Germany.
Update 24 hours later: When checking the logs again I noticed that MegaCrawler had done more than 6000 requests. That is unacceptable. I’m blocking 184.108.40.206 in the .htaccess file.