While examining the access.log at I found around 1000 hits from a bot named operating from the IP address: - - [03/Sep/2019:23:15:50 -0700] "GET /library/file/hkcmd.exe-188/ HTTP/1.1" 200 19377 "-" "Mozilla/5.0 (compatible;; +"

Unfortunately,’s crawler page does not clearly explain why it is a good idea to let the bot crawl my site. Nor does it clearly explain what user User-agent name to use in order to explicitly delay or disallow the bot. In addition to this, it sounds as the maximum value for Crawl-delay is 5 seconds.

I hope that the 5 seconds max value is just a typo. I’m going to try to slow down or block the bot with the following entries in robots.txt:

Crawl-delay: 3600


Disallow: /

I did a reverse IP lookup on and the bot is running at reverse ip lookup

I tried a few geolocation services and all report that the server is located in Germany.

Update 24 hours later: When checking the logs again I noticed that MegaCrawler had done more than 6000 requests. That is unacceptable. I’m blocking in the .htaccess file.