12

My main use-case for using search within the CME is when I'm trying to get a list of templates that have some value in them. My experience so far has been that the SOLR-based search implementation is very much oriented towards finding words in text. When searching for code constructs, they very often don't abide by the word-boundary conventions relevant to free prose. So if I search for "column" I will find a template that contains <div class="column halfWide">

If I search for nav:Navigation I get a message about undefined field nav Unable to get the list of search results - so presumably the colon has special meaning. Even putting my search text in quotes does not suppress this behaviour. If I try to avoid the colon by searching for Navigation it doesn't get found. Similarly - escaping the colon with a backslash does no good.

Prior to testing I have reindexed the system, and I can search successfully for "wordish" content as described above. RTFM does not throw much light on the subject.

  • I believe that SOLR by default should support both techniques you've tried (escaping and quotes) - wiki.apache.org/solr/… presumably Tridion is handling them in a special way. Is the advice in this answer tridion.stackexchange.com/a/617/666 of any help? – Mark Sizer Nov 1 '13 at 8:47
  • @Dom, Check my updated answer. Let me know if that works. – Ram G Nov 1 '13 at 14:30
11
+50

To answer the question on why you can't get results while searching on strings inside quotes or strings which have special characters in them like a : is to go down the Solr rabbit hole and find out why that actually is.

First of all, there are a couple of characters which always have to be escaped when performing a search query. In Solr these are: + - && || ! ( ) { } [ ] ^ " ~ * ? : \

So a search on:

((string.IsNullOrEmpty("FOO") && !(bar)) : 10+32 ? 43-1;

has to become:

\(\(string.IsNullOrEmpty\(\"FOO\"\) \&& \!\(bar\)\) \: 10\+32 \? 43\-1;

.. if you ever want to search for it.

Then there is the question on why you can't search on nav:navigation when you know you have a TBB which for example contains the following:

<style type="text/css" class="nav:navigation"> table{color: #333; text-align: left;</style>

The reason you will not be able to find this string in a template in Tridion, is because Tridion indexes the content of a TBB in a field called ContentText, which is defined in Tridion's Solr configuration schema.xml file, by using Solr's solr.WhitespaceTokenizerFactory, which in turn is defined as a Solr field type called 'text':

<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
     <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ReversedWildcardFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
</fieldType>

The above basically means that upon tokenizing input (aka indexing a search item), the string is split on whitespace and this means that only the values of the splitted string can actually be searched on. So in the above example, you would get a result back if you searched on:

class=\"nav\:navigation\">

.. simply because this string is actually which is the exact token on which can be searched. Also notice another caveat: the < and > characters are sometimes converted to their &lt; and &gt; equivalents, so it is also important to know when to enter the html entity variant if you want to search for the above string. I suspect that it depends whether you for instant search for converted markup inside XML or not.

This also explains why searching for *nav\:navigation* will yield a result, because if the wildcard token (*) is used as prefix and suffix, Solr will be able to match a complete token (in this case class="nav:navigation">), which happends to be surrounded by whitespace.

You can test all the above by going to your Solr instance on tridion:8983/tridion/admin. The other interesting bit you'll see here is that if you turn on highlighting in the Solr admin, you will see that Solr can only highlight the complete token and not substrings of that token.

However

It would be really nice to also be able to search for just "nav"and get back the above result. After a lot of fiddling around with Solr's different tokenizer classes, I came upon a simple solution: simply replace all non alpha numeric characters by whitespace prior to tokenizing content. In this test case it was sufficient, although in the real world additional analyzing and stemming of how to index Tridion content and template code properly is probably necessary. The way to do this is to change the "text" field type of Tridion's Solr configuration:

<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
    <analyzer type="index">
        <charFilter 
            class="solr.PatternReplaceCharFilterFactory"
            pattern="([^a-zA-Z0-9])"
            replacement=" "
            replace="all"/>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ReversedWildcardFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <charFilter 
            class="solr.PatternReplaceCharFilterFactory" 
            pattern="([^a-zA-Z0-9])" replacement=" " 
            replace="all"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
</analyzer>

Running a solr.PatternReplaceCharFilterFactory prior to actually tokenizing content, whereby all non-alphanumeric characters are replaced by whitespace, will do the trick of having only alphanumeric tokens.

If the search host is restarted and the search index is reindexed, you will see that you will be able to just search for nav and get back the proper result. Note that this does not change what is stored in the index, but just what is tokenized. In this example, searching for class=\"nav\:, gives back:

    <style type="text/css" <em>class</em>="<em>nav</em>:navigation"> 
table{color: #333; text-align:left;</style>

and searching for nav gives back:

    <style type="text/css" class="<em>nav</em>:navigation"> 
table{color: #333; text-align:left;</style>

where the strings between the <em> tags are the actual tokens found and thus highlighted, and the token delimiters =," and : are not. And that is exactly what we would need if we were to search for meaningful code fragments in a proper way, so I'll probably always will change the field type configuration on my development machines (only) from now on. :)

  • Thanks Raimond. This is a great answer. It raises as many questions though ...also great! What would the Tridion support position be for modifying the solr config? (You've suggested doing it only on your dev box, perhaps because of this issue.) Would it be interesting to make enhancement requests to get better default behaviour? (I'd think the configuration ought to be different for templates and content, and perhaps different for different types of template.) Would this be a good candidate for a GUI extension/power tool - perhaps injecting a "literal search" box? A well-deserved bounty! – Dominic Cronin Nov 6 '13 at 8:17
  • Thanks Dominic! It would certainly be good to enter an enhancement request, as this would be only a small change to make. To keep backward compatibility and to keep searching for other ItemTypes the same, I would definitely propose to index Templates in a different field type to allow for different tokenizing. This would be easy to do. Further, I would think that SDL will not officially support fiddling with the search index unless it has been thoroughly tested. On a dev instance this would be no problem, I presume. – Raimond Nov 6 '13 at 8:41
  • A literal search box GUI extension would be brilliant, because you would then be able to search while using highlighting, control what you get back in a better way and have more power in general over constructing queries. :) – Raimond Nov 6 '13 at 8:43
6

Tridion CMS using SOLR, : has very special meaning.

SOLR query syntax is to treat : as field name and value (fieldname:value) so the reason why you are getting the error that undefined fieldvalue since nav is not defined as solr schema field.

When you search just "Navigation" solr could not found the text because the word "Navigation" is not separated by a default Tokenizer (white space is typically/standard, could be others as well). Solr indexes the content using Tokenizers to identify the words in the text.. In this case "nav:Navigation" is treated as one word and not indexed the words separately.

Back to your question,

I am aware of SOLR 4 works the way you have mentioned like using quotes or backslash as escape character and CMS does not use SOLR 4 (i guess it is something like SOLR 3 or 3.6). I do not know if we could escape the same way in earlier versions (< 4) of SOL, but worth trying : "nav:Navigation" (OR) "nav\:Navigation"

UPDATE:

I am assuming you have an opening tag "<" before your nav:Navigation so you wont be able to find just by searching nav:navigation as the keyword tokenizer does not match the word alone.

You need to search the whole thing as one word. search with <nav\:Navigation , so SOLR will be able to find this as single word in the index. whether you use double quotes or not use double quotes it does not matter since the whole word is treated as a single word. You still need to use escape character to escape the colon.

  • Thanks Ram, but as I said in my question "escaping the colon with a backslash does no good", although RTFM suggests it should. – Dominic Cronin Oct 30 '13 at 6:01
  • That seems to help quite a lot, but then it still seems to depend on there being some sort of word boundary in front of the "<". Bart's suggestion using "*" seems to work better. I'll test more later. – Dominic Cronin Nov 1 '13 at 16:02
4

As Ram indicated, : is a special character in SOLR (so you need to escape it with a \) and also SOLR indexes the content using tokenizers to identify the words in your text (which will be spaces).

I have the following content in my templates:

<div id="something" xmlns:tridion="http://www.tridion.com/ContentManager/5.0">

And I can search for xmlns:tridion by using the following search statement:

xmlns\:tridion*

So for you to find your nav:Navigation, try one of the following:

nav\:Navigation*

or

*nav\:Navigation*

3

I have just tested on an internal build of Tridion 7.1 (SDL Tridion 2013 Service Pack 1) which is using Lucene 4.3 - which I strongly doubt is the reason for my results, but thought I should mention it - and was able to find TBBs containing <context\:Family.

context\:Family did not work, and neither did context:Family. But including < at the beginning of the search (as Ram correctly identified) did work for me.

PS - I made sure to synchronize my search index before testing this...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.