SOLR Search Engine Integration in asp.net website and Tridion 2011 SP1

Question

I have requirement to integrate the SOLR search engine in my website. As I know Tridion 2011 also use SOLR as the search engine in its content management system, but I am not sure that I can use it for my requirement anyway.

Could anybody help me to provide some in-depth thoughts along with code sample and best practices here? I would like to align myself in right and best direction on same.

I have below question my head at the moment:

1. How SOLR search engine will work with Tridion databases( CM and broker)

2. How content manager and broker database get indexed?

3. How SOLR can be integrated with any Crawler?

4. [WOULD LIKE TO REQUEST TO COMMUNITY TO MAKE THIS LIST OF QUIRES/CONCERNS AS LONG AS BENEFICIAL FOR US ]

Last but not least , This is first time when I am going to integrate any search engine in any website so I would like to request to community for their value addition in form of edit/update here.

Answer 1

Tridion uses SOLR for its search on the Content Manager.

The Content Delivery side doesn't come with an integrated full-text search, but does have extensive structured query capabilities.

In addition many people have integrated it with their preferred search solution. Using a web site crawler is definitely an option, but I expect that others will chime in with their approach.

Answer 2

My 2 cents:

• You cannot use the CM-side search on your website, the fact it uses SOLR means nothing in this context
• I would advice against crawling the delivery database, and instead recommend indexing the rendered pages
• How to integrate SOLR with existing pages is something you could probably ask Google.

What do you want to do with SOLR and Tridion? I know of implementations where the whole Content Delivery stack is running on SOLR (and not on Tridion), and other implementations where SOLR is merely the search engine.

SOLR is a search engine, so my recommendations here also apply.

Answer 3

I don’t think CM-side search can fit in your case anyway as already suggested in above answers.

SOLR also works in very similar pattern as other search engines generally works. Though the above answers have already highlighted major point.

Here I would strongly suggest you to forget that you are using Tridion first and just find out how to do it without Tridion .

In very simple words, SOLR (other search engines) will crawl your site and gather all information it needs and save this information in their database .Now whenever any search will happens in your website they will serve content/URL from their database.

For this as a very first step you will require to configure your own search server /database. It is optional sometimes and depends upon your requirement. Few paid search providers such as “Open Object” maintains own servers for this and just crawls your content/pages and makes entries in their database and serve the search results whenever any end user does any search at your website.This crawling get performed by crawler in place .Crawler are the backbone of any search engine which combined with clever algorithms to work out. You may require to write a deployer extension that notifies SOLR every time something is published/unpublished so that SOLR can update the indices. You have to use JAVA to write this deployer extension. If you are .net guy you can implement your functionality via a WCF service and just by creating a simple client for it from the Java side.

Answer 4

If you are using Solr for your website search, take a look at the open source Search Integration frameowork that has just been released: Search Integration For Tridion (or SI4T). This has an out of the box integration with Apache Solr, but you can use pretty much any search engine you like, with a bit of custom coding to extend the indexing part of it (the templating and storage extension parts are generic to all search engines). Note that currently this framework only supports pushing content directly into the index, but the ambition is to also support crawler notification in the near future.

Answer 5

As other professionals has mentioned, you can not and you should not use the underlying SOLR search engine for Content Manager/Broker Database. If you want to use the broker data you try to explore the OData Service (aka Content Delivery Service).

To use SOLR (or in fact any search engine like Endecca, MS-FAST, Google Mini etc. ) as a search engine for your website and integrate it with SDL Tridion, a detailed generic approach along with High level Architecture has been defined here (keeping in mind the FAQ for this site; the link shared is of my personal blog)

As you are asking for best practice, please consider following scenarios (I am assuming you must have done this much research)

• Are you planning to have a Single Core or Multi-Core SOLR - If it is a multi-core SOLR, then keep your Blueprinting in line with that

• If you are planning to index static components which are embedded in pages as well in SOLR (Which I really don't see a reason unless you are planning to use SOLR itself as a Crawler), then for such components use the Event System to index them in SOLR

Also for SOLR related queries I would recommend posting separate questions going forward in respective communities meant for SOLR, but to close the loop, below are the answers:

1) Regarding SOLR integration with Crawler, you have veriety of crawlers available like Tika and similar which provide integration with much ease - Explore/Research further on SOLR communities

2) If you want to use SOLR as a Web Crawler, you can do so depending on your business requirements and architecture where you are supposed to index all the components with all their fields into SOLR. This approach comes with a disadvantage when you have to deals with trillions of data and with an advantage that from your website you need not to connect with Tridion at all unless for internal Link resolution which will enhance your performance

3) Also, keep in mind that if you are planning to use multi-core SOLR, do not use SolrNet API instead take help of Third Party APIs like StructureMap or Castle Windsor meant for it.

Answer 6

It is important to point out that Solr by itself is not a crawler. It is purely an indexing mechanism based on the well known Lucene format with a couple of APIs to store and retrieve (search) content. Similarly, Apache Tika is not a crawler, but an (sub-Solr) API to store binary content documents (doc, docx, pdf, etc, etc) into a Solr index.

To use Solr on your websites, your implementation must consist of at least three things:

1. A way to get published Tridion content into the search index
2. A proper search index configuration. This is often an underestimated task; to use the real power of a search engine like solr, a lot of tuning is often required
3. An API to let website visitors run searches. Although this may also seem pretty straightforward, the way search results can be retrieved according to business needs will also require some pretty sophisticated tweaking of search queries.

My general recommendations for Solr usage combined with Tridion:

• Although it may be overengineering, as someone mentioned, pushing content into the index by means of a Storage extension gives a lot of fine grained control as to what is pushed into the index in real time. Think for instance of pushing component metadata which will never appear on any page, but should be searchable by website visitors. Pushing content from a deployer extension allows you to do that in clean way; How you store and retrieve that content is indeed dependent on business needs and can be totally tailored to those needs this way;

• I generally recommend against using a crawler like Nutch. This is because you'll have to run regular crawls which can be quite heavy on performance and never will give you real time results in case of content changes. I also found Nutch not reliable enough in terms of processing crawls with a 100% success rate, let alone the unnecessary clutter it produces in your index;

• By using the Storage extension mechanism, you can setup your own fields in your index. These in turn can be tweaked very well in terms of search term boosting, relevancy configuration and other important search features any modern search engine has. I found that when using a crawler that this is a lot harder to do.

So, as you can see and as others have pointed out, implementing search is not a trivial task and requires a lot of research in what you really need in the first place.

.