5

I have requirement to integrate the SOLR search engine in my website. As I know Tridion 2011 also use SOLR as the search engine in its content management system, but I am not sure that I can use it for my requirement anyway.

Could anybody help me to provide some in-depth thoughts along with code sample and best practices here? I would like to align myself in right and best direction on same.

I have below question my head at the moment:

  1. How SOLR search engine will work with Tridion databases( CM and broker)

  2. How content manager and broker database get indexed?

  3. How SOLR can be integrated with any Crawler?

  4. [WOULD LIKE TO REQUEST TO COMMUNITY TO MAKE THIS LIST OF QUIRES/CONCERNS AS LONG AS BENEFICIAL FOR US ]

Last but not least , This is first time when I am going to integrate any search engine in any website so I would like to request to community for their value addition in form of edit/update here.

3
  • 4
    It's great that you asked the question, however it would also be good if you showed what research you've done so far, what has helped and where you got stuck. Apr 1, 2013 at 15:38
  • 1
    Tridion CM search index is not to be used for front end search. You want the front end search index to have only published content.
    – robrtc
    Apr 1, 2013 at 16:52
  • I've found that blog posts, Linked-In, and the old forum as better places to solicit open queries about a particular topic. Consider removing CM from points 1 and 2 and spending time to research your last two points, starting with either your website technology and/or the Solr documentation. Apr 5, 2013 at 4:31

6 Answers 6

8

Tridion uses SOLR for its search on the Content Manager.

The Content Delivery side doesn't come with an integrated full-text search, but does have extensive structured query capabilities.

In addition many people have integrated it with their preferred search solution. Using a web site crawler is definitely an option, but I expect that others will chime in with their approach.

5

My 2 cents:

  • You cannot use the CM-side search on your website, the fact it uses SOLR means nothing in this context
  • I would advice against crawling the delivery database, and instead recommend indexing the rendered pages
  • How to integrate SOLR with existing pages is something you could probably ask Google.

What do you want to do with SOLR and Tridion? I know of implementations where the whole Content Delivery stack is running on SOLR (and not on Tridion), and other implementations where SOLR is merely the search engine.

SOLR is a search engine, so my recommendations here also apply.

5

I don’t think CM-side search can fit in your case anyway as already suggested in above answers.

SOLR also works in very similar pattern as other search engines generally works. Though the above answers have already highlighted major point.

Here I would strongly suggest you to forget that you are using Tridion first and just find out how to do it without Tridion .

In very simple words, SOLR (other search engines) will crawl your site and gather all information it needs and save this information in their database .Now whenever any search will happens in your website they will serve content/URL from their database.

For this as a very first step you will require to configure your own search server /database. It is optional sometimes and depends upon your requirement. Few paid search providers such as “Open Object” maintains own servers for this and just crawls your content/pages and makes entries in their database and serve the search results whenever any end user does any search at your website.This crawling get performed by crawler in place .Crawler are the backbone of any search engine which combined with clever algorithms to work out. You may require to write a deployer extension that notifies SOLR every time something is published/unpublished so that SOLR can update the indices. You have to use JAVA to write this deployer extension. If you are .net guy you can implement your functionality via a WCF service and just by creating a simple client for it from the Java side.

5

If you are using Solr for your website search, take a look at the open source Search Integration frameowork that has just been released: Search Integration For Tridion (or SI4T). This has an out of the box integration with Apache Solr, but you can use pretty much any search engine you like, with a bit of custom coding to extend the indexing part of it (the templating and storage extension parts are generic to all search engines). Note that currently this framework only supports pushing content directly into the index, but the ambition is to also support crawler notification in the near future.

1
  • This is a great solution to your search needs. Please check out the sources and let us know of your results.
    – robrtc
    Oct 22, 2013 at 21:37
4

As other professionals has mentioned, you can not and you should not use the underlying SOLR search engine for Content Manager/Broker Database. If you want to use the broker data you try to explore the OData Service (aka Content Delivery Service).

To use SOLR (or in fact any search engine like Endecca, MS-FAST, Google Mini etc. ) as a search engine for your website and integrate it with SDL Tridion, a detailed generic approach along with High level Architecture has been defined here (keeping in mind the FAQ for this site; the link shared is of my personal blog)

As you are asking for best practice, please consider following scenarios (I am assuming you must have done this much research)

  • Are you planning to have a Single Core or Multi-Core SOLR - If it is a multi-core SOLR, then keep your Blueprinting in line with that

  • If you are planning to index static components which are embedded in pages as well in SOLR (Which I really don't see a reason unless you are planning to use SOLR itself as a Crawler), then for such components use the Event System to index them in SOLR

Also for SOLR related queries I would recommend posting separate questions going forward in respective communities meant for SOLR, but to close the loop, below are the answers:

1) Regarding SOLR integration with Crawler, you have veriety of crawlers available like Tika and similar which provide integration with much ease - Explore/Research further on SOLR communities

2) If you want to use SOLR as a Web Crawler, you can do so depending on your business requirements and architecture where you are supposed to index all the components with all their fields into SOLR. This approach comes with a disadvantage when you have to deals with trillions of data and with an advantage that from your website you need not to connect with Tridion at all unless for internal Link resolution which will enhance your performance

3) Also, keep in mind that if you are planning to use multi-core SOLR, do not use SolrNet API instead take help of Third Party APIs like StructureMap or Castle Windsor meant for it.

4
  • Why do you not recommend to use the SolrNet API in case of multi core configuration?
    – Raimond
    Apr 2, 2013 at 17:49
  • I think the best place for this question is a Solr Forum as it might omit the purpose of this site, however, to give a quick answer for you query: SolrNet API does not support the multi-core configuration completely. The multicore support in SOLRNet is limited to multiple cores with different schema, however, it does not support those multi core scenario where schema is same. In those cases, you will not get a proper solution but as a work around (or probably a hack) that involvles creating absolute multiple redundant copy of your data structure Apr 3, 2013 at 3:20
  • 1
    I asked the question because I think it's not true what you're saying. Further, I don't see having multiple cores with their own configuration and index as hack; It's precisely why Solr supports multiple cores. But let's indeed have this discussion in a Solr environment. ;)
    – Raimond
    Apr 3, 2013 at 7:13
  • The heck I mentioned is related to the client side code you may be needed for creating solr object. What you say is absolutely fine, but using SolrNet while creating objects for multiple core requires a data structure. Now as far as the schema is different (and so is the data structure) this is fine but in case of same schema for two cores and same data structure it will not be possible as Solr object instantiation is a singleton process (and the heck would be to have different data structure in your C# code which actually would be redundant) Apr 3, 2013 at 11:25
2

It is important to point out that Solr by itself is not a crawler. It is purely an indexing mechanism based on the well known Lucene format with a couple of APIs to store and retrieve (search) content. Similarly, Apache Tika is not a crawler, but an (sub-Solr) API to store binary content documents (doc, docx, pdf, etc, etc) into a Solr index.

To use Solr on your websites, your implementation must consist of at least three things:

  1. A way to get published Tridion content into the search index
  2. A proper search index configuration. This is often an underestimated task; to use the real power of a search engine like solr, a lot of tuning is often required
  3. An API to let website visitors run searches. Although this may also seem pretty straightforward, the way search results can be retrieved according to business needs will also require some pretty sophisticated tweaking of search queries.

My general recommendations for Solr usage combined with Tridion:

  • Although it may be overengineering, as someone mentioned, pushing content into the index by means of a Storage extension gives a lot of fine grained control as to what is pushed into the index in real time. Think for instance of pushing component metadata which will never appear on any page, but should be searchable by website visitors. Pushing content from a deployer extension allows you to do that in clean way; How you store and retrieve that content is indeed dependent on business needs and can be totally tailored to those needs this way;

  • I generally recommend against using a crawler like Nutch. This is because you'll have to run regular crawls which can be quite heavy on performance and never will give you real time results in case of content changes. I also found Nutch not reliable enough in terms of processing crawls with a 100% success rate, let alone the unnecessary clutter it produces in your index;

  • By using the Storage extension mechanism, you can setup your own fields in your index. These in turn can be tweaked very well in terms of search term boosting, relevancy configuration and other important search features any modern search engine has. I found that when using a crawler that this is a lot harder to do.

So, as you can see and as others have pointed out, implementing search is not a trivial task and requires a lot of research in what you really need in the first place.

.

4
  • If you are using SDL Tridion 2011, then never use Deployer Extension as it is meant for deployment related activity not with storing data. Secondly it brings following disadvantages: 1) You will not have any control: Assume your publishing is successful but data indexing in SOLR is failed - You will be grilled here and you will not be able to control this behaviour 2) Indexing is related to storing of your data and best place per Tridion architecture to acheive this is the storage layer and you should use Storage Extension for this Apr 3, 2013 at 3:28
  • In addition: If you want to give a more meaningful message in case of failure: for example your publishing is failed because of Solr Indexing is failed or something like that, you can only achieve that by Storage Extension and not by Deployer Extension Apr 3, 2013 at 3:29
  • - 1 for wrong suggestion for using deployer extension in SDL Tridion 2011 for indexing as it will not be nice to deal with storage layer and also you will not be having much control on the process (though in SDL Tridion 2009 it was the only option). Apr 3, 2013 at 3:33
  • @PankajGaur, of course I meant a Storage extension. If indexing fails, you can let the transaction fail, so the page is not published. Sorry for your confusion, but I see deployer extensions nearly synonymous with Storage extensions even though this is technically not the case of course.
    – Raimond
    Apr 3, 2013 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.