Posted by Joe Oldak on

Alas it's the stupid badly behaved web robots that are doing it.

If you look in the stats for your site you'll see that /calendar/view has been viewed 23,697 times in October (so far!).

What happens is that the bad robots ignore the robots.txt and the "nofollow" rules set in the links, and basically browse every possible day in the calendar backwards and forwards ad infinitum. This is of no value to them of course, and yet they do it anyway.

(well behaved robots that follow the rules end up just indexing the events in your calendar but not going to any of the many thousands of possible calendar pages)

I've tried to put things in place to block this behaviour, but ultimately since they look essentially like normal browsers I can't always block it all, because that would also block real users.

Unfortunately this sort of makes the web stats rather less useful than they would otherwise be. :-(

I'll have a think to see if there are any other tricks I'm missing for blocking these robots, but to some extent it seems that this is just the "new normal". Sometimes the Internet really is just a bit of a pain...

Thanks

Joe