If developer-owned, let development handle
robots.txt file as Pankaj suggests.
But authors could control it with a Category:
[ ] Disallow
- Set Robots on Page and/or Structure Group Metadata.
- If different across Publications, move this field to a Component for the page.
- Consider generating the file similar to navigation approaches
If you prefer per-page control, instead use Keywords like:
(x) Everything (INDEX,FOLLOW) [Selected by Default]
( ) Just this page (INDEX,NOFOLLOW)
( ) Just links (NOINDEX,FOLLOW)
( ) None (NOINDEX,NOFOLLOW)
Template the actual text to
I feel managing
robots.txt in the CMS doesn't add much business value and someone might forget it's managed.
Giving editors per-page control in a business-friendly way, with an assumed default, might be more worth the effort.