This is not a question i would say but a discussion point -
While creating a page you very much can have Staging and Live targets to publish them in different environments.
When they are published and are available in open network, they could also be indexed by several search engines. (Google in most of the case)
Two things comes in mind:
1. Why the Dev- Live/Staging & Prod-Staging is out in open network? - This should not be the case, right?
2. If they are open in network in a certain case - How can we ensure only Prod Live sites gets indexed and not others?
AFAIK - DXA does provide a solution of using Page Metadata Schema i.e.
"Search Indexing Metadata" at a page level or include levels viz. Header, Navigation etc.. which allow you to set
"Suppress Content From Search Index:" value checked to stop indexing the page.
For an example - If we are having a single page configured with
"Search Indexing Metadata" and
"Suppress Content From Search Index:" is marked checked then search engine will not crawl through this page, in this case my Live page will also not be indexed which is unacceptable.
How this can be fixed?
REP (robot.txt) is a solution for sure, But my intension here to understand what best we can do from within the CME?