Character Encoding Issue with Si4T and DCPs

Question

I am having an issue with character encoding while indexing DCPs into SOLR using Si4T. DCPs published which make use of the "Enable Search Indexing" building block have their French accent characters encoded improperly in SOLR. However, if the same Component Template is set to be published embedded on a Page instead of being dynamic and that Page is then published with the same Component attached then the entry in SOLR will be correctly encoded.

For example if we have a Component that contains the content "tést Test Unique" and it is published both as a DCP and as a embedded Component on a Page, the following is observed in SOLR when a search for "Test Unique" is completed in the SOLR dashboard:

{
        "pubdate": "2017-04-21T16:03:11.834Z",
        "id": "dcp:2051-11281-23297",
        "parentsgid": [
          0
        ],
        "publicationid": [
          2051
        ],
        "url": [
          "tcm:2051-11281"
        ],
        "body": [
          "tést Test Unique. tést.. "
        ],
        "title": [
          "tést"
        ],
        "schemaid": [
          2820
        ],
        "type": [
          0
        ],
        "itemtype": [
          16
        ],
        "_version_": 1565304387622928400
      },
      {
        "pubdate": "2017-04-21T16:03:10.492Z",
        "id": "tcm:2051-11327-64",
        "sgid": [
          1163,
          1043
        ],
        "parentsgid": [
          1163
        ],
        "publicationid": [
          2051
        ],
        "url": [
          "/fr/projects/test.aspx"
        ],
        "body": [
          "tést Test Unique. tést.. "
        ],
        "title": [
          "tést"
        ],
        "schemaid": [
          2795
        ],
        "type": [
          0
        ],
        "itemtype": [
          64
        ],
        "_version_": 1565304386215739400
      }

Previewing the DCP in Tridion shows index data that looks good. Same with Template Builder. Additionally, the DCP looks good when published to the file system as well.

Answer 1

Encoding problems are almost always about telling lies, albeit sometimes lies of omission. What I mean by this is that when sending data, you do three things:

1. Transform your data into a stream of bytes using an encoding,
2. Transmit the stream of bytes.
3. Tell your client what encoding needs to be used to interpret the bytes.

Sometimes you don't actually communicate #3, and everyone relies on conventions and defaults... and hence lies of omission. Also, sometimes your client has a limited repertoire of encodings, or it's difficult to tell it which to use, so you can end up forced to use an encoding you didn't want to.

What you are seeing in your output is a pretty characteristic tell-tale of multi-byte UTF-8 sequences being interpreted as single byte ASCII characters. The tell-tale is an accented upper case A followed by some other unexpected character. (If you want to know why, this blog will give you a starting point.)

A web page can be made up of several parts, each coming from several different sources (potentially in different encodings), but we usually tell the browser in the response headers what the encoding is for the whole page. If your data is arriving as UTF-8 and is displaying correctly (your non-SOLR scenario), it's likely that the page is being described as UTF-8. The trick is to go step by step and not trust anything.

The beginning of not trusting anything is to wonder whether someone lied to SOLR, or SOLR is lying to you. So the dashboard appears to be expecting ASCII characters (or Latin 1 or whatever) while it's being sent UTF-8. My first bet would be that someone lied to SOLR, and sent a stream of UTF-8 characters but described it as Latin 1. If that happened, SOLR would just innocently send you the same characters to the dashboard. It might even re-encode them as UTF-8 and tell you that you are getting UTF-8, and you'd see the same letters getting rendered.

So if at all possible, use low level tools (such as a byte editor to see what bytes are written to disk, and maybe a network proxy to see what's going over the wire) and establish exactly what the bytes are. Double check at each stage what's getting sent and what the assumed or stated encoding is at both the sending and receiving end.

The other avenue of attack is to double check all your configurations. What encodings are specified in Tridion and in SOLR?

Answer 2

tl;dr to Dominic Cronin's answer, encoding issues in SDL Tridion are always an issue of some system down the publishing/deployment line not being correctly configured.

Read Elena Serghie's blog post 7 Clues to Solve Character Encoding Issues for more details. My initial guess on what is wrong, will be the Deployer JVM settings (as these are by default not on UTF-8). Check out the -DFile.encoding parameter settings in the debug logs of your Deployer.

Answer 3

It turns out that the Deployer was the guilty party. Updating the JVM settings is what fixed it. More information on this can be found here: Tridion 2013 SP1 DD4T SOLR index arabic character encoding issue