0

This is not a question i would say but a discussion point -

While creating a page you very much can have Staging and Live targets to publish them in different environments.

When they are published and are available in open network, they could also be indexed by several search engines. (Google in most of the case)

Two things comes in mind:

1. Why the Dev- Live/Staging & Prod-Staging is out in open network? - This should not be the case, right?

2. If they are open in network in a certain case - How can we ensure only Prod Live sites gets indexed and not others?

AFAIK - DXA does provide a solution of using Page Metadata Schema i.e. "Search Indexing Metadata" at a page level or include levels viz. Header, Navigation etc.. which allow you to set "Suppress Content From Search Index:" value checked to stop indexing the page.

For an example - If we are having a single page configured with "Search Indexing Metadata" and "Suppress Content From Search Index:" is marked checked then search engine will not crawl through this page, in this case my Live page will also not be indexed which is unacceptable.

How this can be fixed?

REP (robot.txt) is a solution for sure, But my intension here to understand what best we can do from within the CME?

Improve this question
6
  • 2
    The easiest is to not expose staging to the Internet, there's no way to control this via the CME, other than through implementation
    – Nuno Linhares ♦
    Commented Apr 28, 2017 at 7:49
  • Hmm that's right.
    – Vikas Kumar
    Commented Apr 28, 2017 at 7:51
  • 1
    and, if for some reason, you must put your staging site on a public server, you can always use robots.txt to specify search engines shouldn't index the site, like you mention in your post. Another option would be to make your staging site password-protected. But, as Nuno said, all this is something you do through implementation, you can't manage this via the CME.
    – Harald Hoffelinck
    Commented Apr 28, 2017 at 10:33
  • 1
    Could you write a TBB to push whatever the Web 8 equivalent is of Publication Target / Target Type in order to write out your metadata conditionally? In 2011 this was accessible via engine.PublishingContext
    – Dave Forber
    Commented Apr 28, 2017 at 12:46
  • 1
    The staging sites can be open to public as in some scenarios people need to access from different locations which makes IP white listing or intranet impractical. You can protect these staging sites with basic authentication (which by the way can be implemented on web server layer. You can get further protection using Akamai's Web Application Firewall. Regarding robot.txt It is indeed the best way not letting Google index it and its content can also be managed in CME
    – Hao Peng
    Commented May 23, 2017 at 17:53

1 Answer 1

Reset to default
-1

So based on the great comments, the answer seems to be 3 fold:

  1. Ideally there is no straight way
  2. Use REP (robot.txt) - Most preferable solution and is widely used.
  3. Write a TBB to deal with it when you publish.

All seems to be reasonable answers.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.