I have a robots.txt exclusion on my website to block crawlers from certain parts of my site. How does this affect your collecting activity?

Answer

UVic Libraries attempts to collect as much of the site as possible in order to create an accurate snapshot for future researchers. We notify site owners before crawling and we generally respect robots.txt exclusions. Please contact us immediately if you have questions about this policy.

  • Last Updated Jul 18, 2025
  • Views 3
  • Answered By Corey Davis

FAQ Actions

Was this helpful? 0 0