1: Surveys Tweaks and Robot Blocking
Posted by Joe Oldak on 08/04/25 16:38

Hi all,

Thanks for the useful Zoom User Group Chat today. A few other bits and bobs to report.

I've made a few changes and fixes in the Surveys tool:

  • Can now embed multi-section surveys on a page using the Survey includelet.
  • Can view and delete individual Anonymous responses (previously could only do this with responses from logged in users)
  • CSV export removes HTML tags from questions

I've also spent quite a bit of time trying to block badly behaved web robots from hitting the site excessively. In particular where they would loop forever in the Calendar going further and further forward and backward in time!

There are more improvements in the pipeline. See the other discussions for input on these :-)

Thanks

Joe

2: Re: Surveys Tweaks and Robot Blocking (response to 1)
Posted by Joe Oldak on 09/04/25 11:29

For anyone interested, the recent bad-robot blocking seems to have had a positive outcome! Last week we were getting around 1.5 million hits a day from badly behaved robots in the various calendar apps around the site. (That's an average of 17 hits per second!)

This week it's down to 50,000 or so per day, and yesterday only about 10,000.

The main benefit here is that it reduces server load, thus making the site faster for real users. It also means the web stats will be a bit more believable!

Note that this is just the bad robots doing this - the well behaved robots (e.g., from Google and Bing) follow the robots.txt rules and use the sitemap.xml files, and so generate only a moderate amount of traffic. These were bad robots just going crazy constantly following every link they could find even though they were excluded in the robots.txt configuration.

Cheers

Joe