Microsoft’s Live search blows. I noticed a few hits from Live.com so I changed my robots.txt file to deny it. Actually, I’ve set it up to deny all crawlers. About two months have passed and I’m still getting the odd referral from Live.com.
It’s dumb because I can see in my server logs that the crawler is checking the robots.txt file, sees that I’m asking it to fuck off, and it fucks off without crawling any pages. So given that the crawler visits at least three times a day, why do they still have my site in their database?
Having had this problem with Google and easily resolving it after a brief message exchange, I searched for the contact information for the Live.com people. I searched in vain. I could leave a comment, but they tell me to not include personally identifiable information, and they’ll certainly not contact me personally. How warm and friendly. So I did leave a comment:
The robots.txt on my site was changed to block all crawlers some two months ago. Your crawlers hit the robots.txt multiple times every day so they certainly know of the change…yet I continue to receive traffic from Live searches. It’s ridiculous that you don’t provide any contact information so we can sort this out.
God forbid you actually let people talk to you!
Ass hats. If they think they’ll beat Google at its own game, they’ve got a good lesson to learn. People love Google. Google will talk to their customers. Microsoft does its best to avoid having to talk to its customers. How they think they’ll earn the same cachet as Google is an utter mystery to me.