added navigation point back
Source Link
Alvin Reyes
  • 12.6k
  • 4
  • 20
  • 61

If developer-owned, let development handle robots.txt file as Pankaj suggests.

But authors could control it with a Category:

Robots
[ ] Disallow

  • Set Robots on Page and/or Structure Group Metadata.
  • If different across Publications, move this field to a Component for the page.
  • Consider generating the file similar to navigation approaches

If you prefer per-page control, instead use Keywords like:

(x) Everything (INDEX,FOLLOW) [Selected by Default]
( ) Just this page (INDEX,NOFOLLOW)
( ) Just links (NOINDEX,FOLLOW)
( ) None (NOINDEX,NOFOLLOW)

Template the actual text to <META NAME="ROBOTS">.

I feel managing robots.txt in the CMS doesn't add much business value and someone might forget it's managed.

Giving editors per-page control in a business-friendly way, with an assumed default, might be more worth the effort.

If developer-owned, let development handle robots.txt file as Pankaj suggests.

But authors could control it with a Category:

Robots
[ ] Disallow

  • Set Robots on Page and/or Structure Group Metadata.
  • If different across Publications, move this field to a Component for the page.

If you prefer per-page control, instead use Keywords like:

(x) Everything (INDEX,FOLLOW) [Selected by Default]
( ) Just this page (INDEX,NOFOLLOW)
( ) Just links (NOINDEX,FOLLOW)
( ) None (NOINDEX,NOFOLLOW)

Template the actual text to <META NAME="ROBOTS">.

I feel managing robots.txt in the CMS doesn't add much business value and someone might forget it's managed.

Giving editors per-page control in a business-friendly way, with an assumed default, might be more worth the effort.

If developer-owned, let development handle robots.txt file as Pankaj suggests.

But authors could control it with a Category:

Robots
[ ] Disallow

  • Set Robots on Page and/or Structure Group Metadata.
  • If different across Publications, move this field to a Component for the page.
  • Consider generating the file similar to navigation approaches

If you prefer per-page control, instead use Keywords like:

(x) Everything (INDEX,FOLLOW) [Selected by Default]
( ) Just this page (INDEX,NOFOLLOW)
( ) Just links (NOINDEX,FOLLOW)
( ) None (NOINDEX,NOFOLLOW)

Template the actual text to <META NAME="ROBOTS">.

I feel managing robots.txt in the CMS doesn't add much business value and someone might forget it's managed.

Giving editors per-page control in a business-friendly way, with an assumed default, might be more worth the effort.

clearer point with better example
Source Link
Alvin Reyes
  • 12.6k
  • 4
  • 20
  • 61

Ram S is rightIf developer-owned, perlet development handle The Web Robots Pages, these go at the navigate-able root of your siterobots.txt file as Pankaj suggests.

In terms of implementation,But authors could control it starts with business requirements and confirming the content model and knowing who owns what parts of the functionality.a Category:

Robots
[ ] Disallow

  • If developer-owned, let development handle the robots.txt file in or outside of Tridion as Pankaj suggestsSet Robots on Page and/or Structure Group Metadata.
  • If the business has a requirement to be able to control disallowed locations (for example)different across Publications, consider generating the file similar to navigation approaches with Tridion. Authors could set metadata on Structure Groups to set move this (not needingfield to a "code" component)Component for the page.

If after you're doneyou prefer per-page control, the business still needs to request development or IT to make the actual changes in almost the same way at the same speed as beforeinstead use Keywords like:

(x) Everything (INDEX, then puttingFOLLOW) [Selected by Default]
( ) Just this inpage (INDEX,NOFOLLOW)
( ) Just links (NOINDEX,FOLLOW)
( ) None (NOINDEX,NOFOLLOW)

Template the CMS isn't adding much business valueactual text to <META NAME="ROBOTS">.

I feel managing robots.txt in the CMS doesn't add much business value and someone might forget it's managed.

Giving editors per-page control in a business-friendly way, with an assumed default, might be more worth the effort.

Ram S is right, per The Web Robots Pages, these go at the navigate-able root of your site.

In terms of implementation, it starts with business requirements and confirming the content model and knowing who owns what parts of the functionality.

  • If developer-owned, let development handle the robots.txt file in or outside of Tridion as Pankaj suggests.
  • If the business has a requirement to be able to control disallowed locations (for example), consider generating the file similar to navigation approaches with Tridion. Authors could set metadata on Structure Groups to set this (not needing a "code" component).

If after you're done, the business still needs to request development or IT to make the actual changes in almost the same way at the same speed as before, then putting this in the CMS isn't adding much business value.

If developer-owned, let development handle robots.txt file as Pankaj suggests.

But authors could control it with a Category:

Robots
[ ] Disallow

  • Set Robots on Page and/or Structure Group Metadata.
  • If different across Publications, move this field to a Component for the page.

If you prefer per-page control, instead use Keywords like:

(x) Everything (INDEX,FOLLOW) [Selected by Default]
( ) Just this page (INDEX,NOFOLLOW)
( ) Just links (NOINDEX,FOLLOW)
( ) None (NOINDEX,NOFOLLOW)

Template the actual text to <META NAME="ROBOTS">.

I feel managing robots.txt in the CMS doesn't add much business value and someone might forget it's managed.

Giving editors per-page control in a business-friendly way, with an assumed default, might be more worth the effort.

Source Link
Alvin Reyes
  • 12.6k
  • 4
  • 20
  • 61

Ram S is right, per The Web Robots Pages, these go at the navigate-able root of your site.

In terms of implementation, it starts with business requirements and confirming the content model and knowing who owns what parts of the functionality.

  • If developer-owned, let development handle the robots.txt file in or outside of Tridion as Pankaj suggests.
  • If the business has a requirement to be able to control disallowed locations (for example), consider generating the file similar to navigation approaches with Tridion. Authors could set metadata on Structure Groups to set this (not needing a "code" component).

If after you're done, the business still needs to request development or IT to make the actual changes in almost the same way at the same speed as before, then putting this in the CMS isn't adding much business value.