Anybody who visited my site during the last week of March would have found that it was temporarily down as it exceeded the monthly bandwidth limit. Not that my site suddenly got popular overnight, nor is it due to any digg effect or stumbling upon. It’s due to a monster called MSNBot coming to my site and sucking away all my bandwidth in a span of two and a half days. All this happened between March 21st and 23rd, unfortunately at a time when I was away and could not connect to the internet. I was not in a position to take timely action and stop the avalanche.
65.55.107.16 - - [24/Mar/2008:21:42:15 -0500] "GET /wordpress/ HTTP/1.1" 500 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
That’s the signature MSNBot left in my access logs. Imagine the same entry repeated over and over and over again for tens of thousands of lines in the log file. The whois info for IP 65.55.107.16 confirms that it’s from the Microsoft Corp, but reverse dns returns bl1sch4081307.phx.gbl (see also this post at seoroundtable.com).
I straight away blocked MSN Bot from further accessing my website. Only about 10% of the total traffic to my website is from MSN and Live search and I don’t mind losing that traffic. Here’s how to do the block just in case you want to know…
<Limit GET POST>
Order Allow, Deny
Deny from 65.55.107.16
Allow from all
</Limit>
Add the above lines in your .htaccess file. This will block the IP 65.55.107.16 from accessing the site. Replace 65.55.107.16 with 65.55. to block the entire IP range.
We all know about MS’s buggy ‘blue screen’ software and their unethical business practices to destroy competitors and monopolise the market, but why would MS target innocent webmasters like me? There are stories around the web that Microsoft Live.com Plays Referrer Stats Games and Ruins Your AdSense Income, though in my case, they did not send any referrer spam, just that their bot ran wild across my little site like a mad elephant.
More stories about MSNBot and Microsoft Live from the web
Leave a Reply