Implementing Robots.txt

Question

What is the best way to implement robots.txt in SDL Tridion?

Is it necessary to keep in the root folder or can we put it in another folder in the site and do a redirect so the Google Crawler can follow and fetch it?

Answer 1

Yes , it is necessary to keep robots.txt in the root folder of your website. You can not put it in any subdirectory of the website for example –

http://www.yourdomain.com/robots.txt is a valid location.

And

http://www.yourdomain.com/home/robots.txt is not valid location.

you can not put this in a subdirectory because bots only check for this file in the root of the domain.

Answer 2

If you ask me, I would never suggest keeping something in SDL Tridion CMS which is not a content (please explore basic CMS concept to learn what really is a content), as CMS is meant for effectively keeping your content and not everything which is being required by your site.

I would suggest the deployment of robot.txt should be out of the CMS, if you have a requirement of keeping or managing robot.txt in CMS, please re-verify the requirement - keep in mind CMS is meant to manage only the content.

I have even seen people keeping DLLs in Tridion and publishing as multimedia component to cut on the deployment - To me it is absolute non-sense.

Golden Words (I learn from a CMS expert) - CMS is not everything in your website or application, it is a very small part of the architecture of your application.

In the extreme case, even if you want to keep robot.txt in CMS, you have to keep it at the root of your website and you may follow one of the following process:

1. Keep robot.txt as a multimedia component and us the C# TBB as explained in earlier post to directly publish it to the root of your website

2. Create a Schema -> Component containing content of robot.txt -> corresponding CT to generate plain text -> corresponding PT to out put as robot.txt -> A page at root structure group.

All comes with their pros & cons.

I hope it helps.

Answer 3

If developer-owned, let development handle robots.txt file as Pankaj suggests.

But authors could control it with a Category:

Robots
[ ] Disallow

• Set Robots on Page and/or Structure Group Metadata.
• If different across Publications, move this field to a Component for the page.
• Consider generating the file similar to navigation approaches

If you prefer per-page control, instead use Keywords like:

(x) Everything (INDEX,FOLLOW) [Selected by Default]
( ) Just this page (INDEX,NOFOLLOW)
( ) Just links (NOINDEX,FOLLOW)
( ) None (NOINDEX,NOFOLLOW)

Template the actual text to <META NAME="ROBOTS">.

I feel managing robots.txt in the CMS doesn't add much business value and someone might forget it's managed.

Giving editors per-page control in a business-friendly way, with an assumed default, might be more worth the effort.

Answer 4

There are probably hundreds of ways to implement things like robots.txt.

Let me start by asking... How would you do it without Tridion? Furthermore, does the robots.txt file need to be content managed?

My guess is that without a CMS, you would probably write a text file in your preferred text editor, add it to your source control system and deploy it out to your site. This approach is still perfectly valid. After all, why do authors/editors need to modify the robots.txt? How often do you intend to change it?

If you still think you need to store it in the CMS, read on...

You could store it in Tridion as a multimedia component and publish it to the root structure group using a .Net based TBB and one of the engine.addBinary() methods. There is a TBB that already exists somewhere on Tridion World that publishes multimedia components from a known folder to a matching structure group (see AppStore).

Another option is to store the content of your robots.txt file using a very simple schema, i.e. a single multi-line plain text field or similar and then output it using a simple page and component template that outputs the text field. This setup can be using for writing other plain text assets, like small chunks of JavaScript code into a page, for example, but I'd advise you to hide this schema from editors (via security) as in my experience they may abuse it to get content onto the site that they would normally be unable to.