ECL Errors Impacting Entire Instance

Question

I am trying to run down a few ECL errors we are encountering from an ECL connector in Web 8 connected to an S3 bucket mountpoint.

It is valid to recognize that we are in a multiple mountpoint ECL scenario.

Basically, when we begin to open a Component that has an association to a Multimedia Component (that resides in a S3 bucket) we notice severe performance latency, to the point until we need to bounce the SDL Web app.

From a user perspective, it's almost like the UI is trying to play catch up after a few mins or items saved & closed. A user can go in and perform basic editorial activity updates in approx ~2-3 secs. I wonder if that is too fast for the UI? After a few mins, it seems to lag, then there are 3-4 tabs opened on the CME for Components still attempting to save and close out.

I have adjusted the timeout settings in TcmServiceHost.exe.config from "̶0̶0̶:̶1̶0̶:̶0̶0̶"  into "01:00:00"

<netTcpBinding>
   ...
    <binding name="StreamDownload_netTcpBinding" maxReceivedMessageSize="2147483647" transferMode="StreamedResponse" sendTimeout="01:00:00"/>
    <binding name="StreamUpload_netTcpBinding" maxReceivedMessageSize="2147483647" transferMode="StreamedRequest" receiveTimeout="01:00:00"/>
    <binding name="TmService_netTcpBinding" transactionFlow="false" transactionProtocol="WSAtomicTransaction11" maxReceivedMessageSize="10485760" receiveTimeout="01:00:00" sendTimeout="01:00:00"><readerQuotas maxArrayLength="10485760" maxStringContentLength="10485760"/></binding>
    <binding name="EclService_netTcpBinding" transactionFlow="true" transactionProtocol="WSAtomicTransaction11" maxReceivedMessageSize="10485760" receiveTimeout="01:00:00" sendTimeout="01:00:00"><readerQuotas maxArrayLength="10485760" maxStringContentLength="10485760"/></binding>
</netTcpBinding>

So I am surprised to still get this error

This request operation sent to net.tcp://localhost:2660/ExternalContentLibrary/2012/netTcp did not receive a reply within the configured timeout (00:10:00)

Windows Application Errors:

Tridion.ExternalContentLibrary.UI.Model.Services.ErrorHandlerBehaviorAttribute HandleError

System.Exception: Unable to load the item ---> System.TimeoutException: This request operation sent to net.tcp://localhost:2660/ExternalContentLibrary/2012/netTcp did not receive a reply within the configured timeout (00:10:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

Server stack trace: at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Tridion.ExternalContentLibrary.Service.Client.ISessionAwareEclService.GetItem(String itemUri, Int32 openMode) at Tridion.ExternalContentLibrary.UI.Model.Services.GeneralImpl.GetItem(SessionAwareEclServiceClient eclClient, String id, Int32 openMode) at Tridion.ExternalContentLibrary.UI.Model.Services.GeneralImpl.GetItem(String id, Int32 openMode) --- End of inner exception stack trace --- at Tridion.ExternalContentLibrary.UI.Model.Services.GeneralImpl.GetItem(String id, Int32 openMode) at SyncInvokeGetItem(Object , Object[] , Object[] ) at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) at Tridion.Web.UI.Core.Extensibility.DataExtenderOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc) at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

Tridion.ExternalContentLibrary.UI.Model.Services.ErrorHandlerBehaviorAttribute.HandleError

Tridion.ExternalContentLibrary.UI.DataExtenders.UriUpdaters.XmlUpdater ProcessResponse

System.TimeoutException: This request operation sent to net.tcp://localhost:2660/ExternalContentLibrary/2012/netTcp did not receive a reply within the configured timeout (00:10:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

Server stack trace: at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Tridion.ExternalContentLibrary.Service.Client.ISessionAwareEclService.GetEclItemTitlesFromTcmUri(IEnumerable1 tcmItemUris) at Tridion.ExternalContentLibrary.UI.DataExtenders.UriUpdaters.UriUpdaterBase.TransformTcmUrisToEclUris(XElement contentElement, PipelineContext pipeline) at Tridion.ExternalContentLibrary.UI.DataExtenders.UriUpdaters.XmlUpdater.ProcessItemXml(XmlTextReader xReader, PipelineContext pipelineContext, Predicate1 isElementToProcess, Action`2 updateElement) at Tridion.ExternalContentLibrary.UI.DataExtenders.UriUpdaters.UriUpdaterBase.ProcessCallResponse(XmlTextReader xReader, PipelineContext pipelineContext) at Tridion.ExternalContentLibrary.UI.DataExtenders.UriUpdaters.XmlUpdater.ProcessResponse(XmlTextReader reader, PipelineContext context) Tridion.ExternalContentLibrary.UI.DataExtenders.UriUpdaters.XmlUpdater.ProcessResponse

Tridion.ExternalContentLibrary.UI.Http.ImageHandler ProcessRequest

System.Threading.ThreadAbortException: Thread was being aborted. at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Tridion.ExternalContentLibrary.Service.Client.ISessionAwareEclService.GetItem(String itemUri, Int32 openMode) at Tridion.ExternalContentLibrary.UI.Http.ImageHandler.ProcessRequest(HttpContext context)

Tridion.ExternalContentLibrary.UI.Http.ImageHandler.ProcessRequest

UPDATE

For this error below, we have identified that the ECL tag in the source of the Component with the S3 association to the mountpoint contains a > 260 character count.

For example,

ecl:301-s3-abcdefghijklmnopqrstuvwxyz!2D;assets!2F;abcdefghijklmnopqrstuvwxyz!2F;things!2D;to!2D;do!2F;abcdefghijklmnopqrstuvwxyz!2F;abcdefghijklmnopqrstuvwxyz!2D;california!2D;adventure!2F;food!2D;and!2D;wine!2D;festival!2D;abcdefghijklmnopqrstuvwxyz!2F;2017!2F;dca!2D;food!2D;and!2D;wine!2D;festival!2D;marketplaces!2D;bacon!2D;abcdefghijklmnopqrstuvwxyz!2D;smoked!2D;bacon!2D;mac!2D;n!2D;cheese!2D;1x1.jpg-fls-file

Tridion.ExternalContentLibrary.UI.Http.ThumbnailCache StoreImage

System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters. at System.IO.PathHelper.GetFullPathName() at System.IO.Path.LegacyNormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.File.InternalWriteAllBytes(String path, Byte[] bytes, Boolean checkHost) at Tridion.ExternalContentLibrary.UI.Http.ThumbnailCache.WriteDataToFile(String path, Byte[] dataToCache) at Tridion.ExternalContentLibrary.UI.Http.ThumbnailCache.StoreImage(Byte[] image, IEclUri eclUri, Int32 maxWidth, Int32 maxHeight, String thumbnailETag, String state, String addInNameAndVersion)

Tridion.ExternalContentLibrary.UI.Http.ThumbnailCache.StoreImage

Seeing that after updating a few items, that the saveitem call begins to hang.

Answer 1

One important point to understand is that ECL mountpoints make a connection to an external system, this will always introduce latency and thus can have a huge effect on the performance of the UI. For each mountpoint there will be multiple connections made to the external system, when browsing through the tree of ECL items and when opening ECL Components.

From the timeout you are getting, it seems you don't have an ideal connection from the CMS to the S3 bucket, or there are too many connections being made at which point the S3 server might be refusing them or stalling them. The timeouts are set by default at 10 minutes, increasing them to 1 hour is not a good solution, if responses are not getting back within 10 minutes, you need to look at the underlying cause for those timeouts, rather than ignoring them. A user will have closed his browser window long before the 10 minute timout occurs, but the underlying action will still wait for a response. Which just increases the problem (since the user most likely will have opened a new browser window and tried the action again, opening yet another set of connections).

Maybe the ECL provider you are using is not cleaning up its connections, or maybe it is just not performing well in general.

Another remark I can make about the S3 ECL connector is that it is using a rather odd way of specifying a unique ID. Unfortunately the S3 API does not supply a unique ID for the items, it seems the only way to uniquely identify an item is through its path. Since ECL items need a unique item ID, the creator of the S3 ECL connector, decided to use the full path of the item as its unique item ID. This results in endless long strings used for the ECL URI, which again is not ideal. To properly use this ECL connector, I suggest making sure your paths are not too long in your S3 bucket. Keep it to a maximum of 3 folders deep and use short folder names.

Better would be to modify the S3 ECL provider and create a mapping table with proper unique IDs for each item, mapping to the paths, but this will require an additional service and database to be integrated in the connection.