7

What is the best way to implement robots.txt in SDL Tridion?

Is it necessary to keep in the root folder or can we put it in another folder in the site and do a redirect so the Google Crawler can follow and fetch it?

  • I think it's implied, but just to be, this is with Tridion? – Alvin Reyes Apr 5 '13 at 20:07
  • 1
    Yes - it's implied. Otherwise the question would be off-topic! – Dominic Cronin Apr 6 '13 at 15:46
8

Yes , it is necessary to keep robots.txt in the root folder of your website. You can not put it in any subdirectory of the website for example –

http://www.yourdomain.com/robots.txt is a valid location.

And

http://www.yourdomain.com/home/robots.txt is not valid location.

you can not put this in a subdirectory because bots only check for this file in the root of the domain.

7

If you ask me, I would never suggest keeping something in SDL Tridion CMS which is not a content (please explore basic CMS concept to learn what really is a content), as CMS is meant for effectively keeping your content and not everything which is being required by your site.

I would suggest the deployment of robot.txt should be out of the CMS, if you have a requirement of keeping or managing robot.txt in CMS, please re-verify the requirement - keep in mind CMS is meant to manage only the content.

I have even seen people keeping DLLs in Tridion and publishing as multimedia component to cut on the deployment - To me it is absolute non-sense.

Golden Words (I learn from a CMS expert) - CMS is not everything in your website or application, it is a very small part of the architecture of your application.

In the extreme case, even if you want to keep robot.txt in CMS, you have to keep it at the root of your website and you may follow one of the following process:

  1. Keep robot.txt as a multimedia component and us the C# TBB as explained in earlier post to directly publish it to the root of your website

  2. Create a Schema -> Component containing content of robot.txt -> corresponding CT to generate plain text -> corresponding PT to out put as robot.txt -> A page at root structure group.

All comes with their pros & cons.

I hope it helps.

  • Very much agree with that, I think too often a CMS is seen as a deployment tool which it really is not (apart from all you can do). But with that said, when people do make it a requirement that a robots.txt is managed inside the CMS, then it is good to know how you can do it. – Bart Koopman Apr 5 '13 at 8:12
  • Point taken, updating the answer :) – Pankaj Gaur Apr 5 '13 at 8:31
  • I don't see why you shouldn't deploy your robots.txt from the CMS. You can argue that it's not content (I'd disagree) but even then it doesn't get us very far. Have you got a concrete reason why it's a bad idea? – Dominic Cronin Apr 5 '13 at 13:22
  • 3
    @PankajGaur I'm aware of many software architecture principles. You still haven't given a reason why we shouldn't manage robots.txt in the CMS – Dominic Cronin Apr 6 '13 at 12:12
  • 2
    @AlvinReyes Should I stop publishing code out of the CMS? Now you're being silly. Should I stop publishing assets that are owned by development? Of course not. The point of a WCMS is that it allows you to manage both those concerns, but differently. If all I want to publish is content, I'll use FTP! – Dominic Cronin Apr 6 '13 at 12:16
3

If developer-owned, let development handle robots.txt file as Pankaj suggests.

But authors could control it with a Category:

Robots
[ ] Disallow

  • Set Robots on Page and/or Structure Group Metadata.
  • If different across Publications, move this field to a Component for the page.
  • Consider generating the file similar to navigation approaches

If you prefer per-page control, instead use Keywords like:

(x) Everything (INDEX,FOLLOW) [Selected by Default]
( ) Just this page (INDEX,NOFOLLOW)
( ) Just links (NOINDEX,FOLLOW)
( ) None (NOINDEX,NOFOLLOW)

Template the actual text to <META NAME="ROBOTS">.

I feel managing robots.txt in the CMS doesn't add much business value and someone might forget it's managed.

Giving editors per-page control in a business-friendly way, with an assumed default, might be more worth the effort.

  • I completely agree with this. "Generating the file similar to navigation approaches" (rather than being editable in a Component) would also ensure that the robots.txt file has the correct syntax. – Jonathan Williams Apr 8 '13 at 16:37
  • 1
    You know it's a post by Alvin when you see simulated check-boxes or radio buttons :D – David Forster Feb 9 '15 at 8:56
  • SEO-friendly and accessible? Or maybe I'm lazy. ;-) – Alvin Reyes Feb 9 '15 at 14:05
2

There are probably hundreds of ways to implement things like robots.txt.

Let me start by asking... How would you do it without Tridion? Furthermore, does the robots.txt file need to be content managed?

My guess is that without a CMS, you would probably write a text file in your preferred text editor, add it to your source control system and deploy it out to your site. This approach is still perfectly valid. After all, why do authors/editors need to modify the robots.txt? How often do you intend to change it?

If you still think you need to store it in the CMS, read on...

You could store it in Tridion as a multimedia component and publish it to the root structure group using a .Net based TBB and one of the engine.addBinary() methods. There is a TBB that already exists somewhere on Tridion World that publishes multimedia components from a known folder to a matching structure group.

Another option is to store the content of your robots.txt file using a very simple schema, i.e. a single multi-line plain text field or similar and then output it using a simple page and component template that outputs the text field. This setup can be using for writing other plain text assets, like small chunks of JavaScript code into a page, for example, but I'd advise you to hide this schema from editors (via security) as in my experience they may abuse it to get content onto the site that they would normally be unable to.

  • 1
    the TBB you mention on World is here sdltridionworld.com/community/extension_overview/… (it's #5: Get CSS Images) and I've written an explanation about the process in this answer before stackoverflow.com/questions/11433013/… – Bart Koopman Apr 5 '13 at 7:04
  • Thanks Bart. At the time of writing my answer I was on my way out so I didn't have time to find it :) – David Forster Apr 5 '13 at 8:02
  • No problem, just figured I'd add the link. One more comment I would like to make on it is that I would favor the process of the Schema and managing it as text. Since managing a text file as binary content is just wrong in my opinion. – Bart Koopman Apr 5 '13 at 8:16
  • 1
    Just to be on the safe side: when you say "hide this schema from editors (via security)" that's open to misinterpretation. If the templating emits verbatim content, you need to ensure that they can not use the schema to make a component. So for example, preventing navigation to the containing folder wouldn't be enough. I'm sure that's what you meant, but let's just be pedantic about it! :-) – Dominic Cronin Apr 5 '13 at 13:26
  • 1
    I think Dominic means removing read from the folder that has the schema can prevent authors from making components based on that schema. Less pendatry, more action-able steps! :-p – Alvin Reyes Apr 5 '13 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.