After upgrading to Web 8.5 from Web 8.1 restarting Content Manager Service Host to fix frequent errors in CME

Question

We have upgraded CMS from Web 8.1 to Web 8.5 couple of months back. We got three CMS nodes behind the F5, We frequently used to see below list of issues and restarting Content Manager Service Host solves the problem and may happen again after 1 week in that node. At least one node per week shows us this behavior.

List of issues:

1. Unable to see thumbnails of images inside components.

2. Used to see below error while page previews or version history comparison etc., in CME.

   (80040356) Could not load file or assembly '0 bytes loaded from Tridion.ContentManager.Templating, Version=8.5.0.466, Culture=neutral, PublicKeyToken=360aac4d3354074b' or one of its dependencies. An attempt was made to load a program with an incorrect format. Bad IL format.

   Bad IL format.

3. Unable to see thumbnails in CME media repository (which are ECL from S3)

We are not seeing any bad CPU or memory metrics in that particular node at the time of failure.

So i am looking for some guidance on how to investigate this?

Here is the log from event viewer:

Could not load file or assembly '0 bytes loaded from Tridion.ContentManager.Templating, Version=8.5.0.466, Culture=neutral, PublicKeyToken=360aac4d3354074b' or one of its dependencies. An attempt was made to load a program with an incorrect format.
Bad IL format.

Component: Tridion.ContentManager.CoreService
Errorcode: 0
User: NT AUTHORITY\NETWORK SERVICE

StackTrace Information Details:
   at System.Reflection.RuntimeAssembly.nLoadImage(Byte[] rawAssembly, Byte[] rawSymbolStore, Evidence evidence, StackCrawlMark& stackMark, Boolean fIntrospection, Boolean fSkipIntegrityCheck, SecurityContextSource securityContextSource)
   at System.Reflection.Assembly.Load(Byte[] rawAssembly)
   at Tridion.ContentManager.Templating.TemplateUtilities.LoadAssembly(Engine engine, Byte[] assemblyContent, String assemblyFileName)
   at Tridion.ContentManager.AssemblyCache.AssemblyCache.<>c__DisplayClass12_0.<GetOrAddAssemblyCacheInfo>b__0(String _)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Tridion.ContentManager.AssemblyCache.AssemblyCache.GetOrAddAssemblyCacheInfo(String script, Func`1 getAssembly, String keyAddition)
   at Tridion.ContentManager.AssemblyCache.AssemblyCache.GetInstanceOfType[T](String script, Func`1 generateAssembly, String typeName, Func`2 getType, String keyAddition)
   at Tridion.ContentManager.Templating.Assembly.CSharpSourceCodeMediator.Transform(Engine engine, Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.ExecuteTemplate(Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.InvokeTemplate(Package package, TemplateInvocation templateInvocation, Template template)
   at Tridion.ContentManager.Templating.Compound.CompoundTemplateMediator.Transform(Engine engine, Template templateToTransform, Package package)
   at Tridion.ContentManager.Templating.Engine.ExecuteTemplate(Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.InvokeTemplate(Package package, TemplateInvocation templateInvocation, Template template)
   at Tridion.ContentManager.Templating.Engine.TransformPackage(Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.TransformItem(Template template, IdentifiableObject itemToRender)
   at Tridion.ContentManager.Templating.TemplatingRenderer.Render(ResolvedItem resolvedItem, PublishInstruction instruction, PublicationTarget target, RenderedItem renderedItem, RenderContext renderContext)
   at Tridion.ContentManager.Publishing.Rendering.RenderEngine.Render(ResolvedItem resolvedItem, PublishInstruction instruction, PublicationTarget target, RenderContext context)
   at Tridion.ContentManager.Publishing.Rendering.RenderEngine.Render(IdentifiableObject item, Template template, PublishInstruction instruction, PublicationTarget target, RenderContext context)
   at Tridion.ContentManager.Publishing.Rendering.RenderEngine.Render(IdentifiableObject item, Template template, PublishInstruction instruction, PublicationTarget target)
   at Tridion.ContentManager.CoreService.CoreServiceBase.RenderItem(TcmUri itemId, IdentifiableObjectData itemDeltaData, TcmUri templateId, TemplateData templateDeltaData, PublishInstructionData publishInstruction, String publicationTargetIdOrPurpose)
   at Tridion.ContentManager.CoreService.CoreServiceBase.RenderItem(String itemId, String templateId, PublishInstructionData publishInstruction, String publicationTargetIdOrPurpose)
   at SyncInvokeRenderItem(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at Tridion.ContentManager.CoreService.CoreServiceInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

Answer 1

• Could be a problem with 32bit/64bit platform loading?

• In the server is there any scheduled job script anything running on a weekly basis?

• Is there any antivirus software installed on the server could be blocking the images folders by scanning?

• Is there any application monitoring tool running on the server?

• Review the F5 firewall settings

• Another thought that crossed my mind is that you might have a dependency on an assembly in the GAC that is missing on your environment, maybe worth reupload this DLL to assembly?

• Tridion.ContentManager.Templating.dll - could be a chance maybe DLL got locked, you need to find this DLL and click Unblock from the context menu

• Review your Event Viewer Application & Tridion logs to see if there are any errors or warnings.