4

We are using SDL Web 8.5 and DD4T 2.0, but we face one problem that our server will get very high CPU and memory usage and the IIS response time get longer at the off-peak time.

We collect dump of w3wp.exe and part of it says some warning about System.Collections.Generic.Dictionary

The following threads in w3wp.DMP are enumerating a System.Collections.Generic.Dictionary object

(25 27 28 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 )

38.16% of threads blocked (29 threads)

Multiple threads enumerating through a collection is intrinsically not a thread-safe procedure. If the dictionary object accessed by these threads is declared as static then the threads can go in an infinite loop while trying to enumerate the dictionary if one of the threads writes to the dictionary while the other threads are reading\enumerating through the same dictionary. You may also experience High CPU during this stage. For more details refer to High CPU in .NET app using a static Generic.Dictionary

We check “Top 5 Threads by CPU time” section on dump file, one of the thread detail as below

Thread 55 - System ID 9052

Entry point   clr!Thread::intermediateThreadProc 
Create time   2018/3/29 銝�� 07:31:48 
Time spent in user mode   0 Days 02:00:53.921 
Time spent in kernel mode   0 Days 00:00:00.640 

This thread is enumerating a System.Collections.Generic.Dictionary object

.NET Call Stack

mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon)+106 
DD4T.Serialization.XmlSerializerService.GetXmlSerializer[[System.__Canon, mscorlib]]()+5a 
DD4T.Serialization.XmlSerializerService.Deserialize[[System.__Canon, mscorlib]](System.String)+279 
DD4T.Factories.ComponentPresentationFactory.TryGetComponentPresentation(DD4T.ContentModel.IComponentPresentation ByRef, System.String, System.String)+279 
DD4T.Factories.ComponentPresentationFactory.b__14_0(System.String)+2d 
System_Core_ni!System.Linq.Enumerable+WhereSelectArrayIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()+4b 
System_Core_ni!System.Linq.Enumerable+WhereEnumerableIterator`1[[System.__Canon, mscorlib]].MoveNext()+e9 
mscorlib_ni!System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1)+1b4 
System_Core_ni!System.Linq.Enumerable.ToList[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1)+46 
DD4T.Factories.ComponentPresentationFactory.FindComponentPresentations(DD4T.ContentModel.Querying.IQuery)+165 
Web.Public.ComponentFactoryExtender.FindValidComponents(DD4T.ContentModel.Factories.IComponentPresentationFactory, DD4T.ContentModel.Querying.IQuery)+56 
Web.Public.Controllers.B2BTWNController.shutdown(System.String)+1ad 
DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, System.Web.Mvc.ControllerBase, System.Object[])+8b 
System.Web.Mvc.ReflectedActionDescriptor.Execute(System.Web.Mvc.ControllerContext, System.Collections.Generic.IDictionary`2)+e5 
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(System.Web.Mvc.ControllerContext, System.Web.Mvc.ActionDescriptor, System.Collections.Generic.IDictionary`2)+23 
System.Web.Mvc.Async.AsyncControllerActionInvoker.b__39(System.IAsyncResult, ActionInvocation)+27 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`2[[System.__Canon, mscorlib],[System.Web.Mvc.Async.AsyncControllerActionInvoker+ActionInvocation, System.Web.Mvc]].CallEndDelegate(System.IAsyncResult)+43 
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(System.IAsyncResult)+2a 
System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters.b__3d()+48 
System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters+<>c__DisplayClass46.b__3f()+2b 
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(System.IAsyncResult)+2a 
System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21+<>c__DisplayClass2b.b__1c()+26 
System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21.b__1e(System.IAsyncResult)+24 
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(System.IAsyncResult)+26 
System.Web.Mvc.Controller.b__1d(System.IAsyncResult, ExecuteCoreState)+1d 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.Controller+ExecuteCoreState, System.Web.Mvc]].CallEndDelegate(System.IAsyncResult)+41 
System.Web.Mvc.Controller.EndExecuteCore(System.IAsyncResult)+34 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.__Canon, mscorlib]].CallEndDelegate(System.IAsyncResult)+24 
System.Web.Mvc.Controller.EndExecute(System.IAsyncResult)+26 
System.Web.Mvc.MvcHandler.b__5(System.IAsyncResult, ProcessRequestState)+2b 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.MvcHandler+ProcessRequestState, System.Web.Mvc]].CallEndDelegate(System.IAsyncResult)+41 
System.Web.Mvc.MvcHandler.EndProcessRequest(System.IAsyncResult)+26 
System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+25f 
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+86 
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+570 
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+79 
System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+1e3 
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+408 
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13 
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+5a 
System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+7e 
[[InlinedCallFrame] (System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion)] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatusByRef) 
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+5df 
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+13 
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+5a 
[[ContextTransitionFrame]]

And we check the link High CPU in .NET app using a static Generic.Dictionary Which describe some reason about high CPU and FindEntry method. Part of content as below:

What is happening here, and causing the high CPU is that the FindEntry method walks through the dictionary, trying to find the key. If multiple threads are doing this at the same time, especially if the dictionary is modified in the meantime you may end up in an infinite loop in FindEntry causing the high CPU behavior and the process may hang.

And we check the code from DD4T, GetXmlSerializer method has _xmlSerializers variable which is a static Dictionary variable. It will use ContainsKey function to FindEntry and it will add new data.

private static Dictionary<Type, XmlSerializer> _xmlSerializers = new Dictionary<Type, XmlSerializer>();
    private XmlSerializer GetXmlSerializer<T>() where T : XmlSerializer
    {
    if (!_xmlSerializers.ContainsKey(typeof(T)))
    {
    XmlSerializer serializer = (T)Activator.CreateInstance(typeof(T));
    _xmlSerializers.Add(typeof(T), serializer);
    }
    return _xmlSerializers[typeof(T)];
    }

We are not sure is it the reason that the GetXmlSerializer function let us face this problem? Does anyone faced similar issue earlier?

1 Answer 1

1

It looks like you are facing a concurrency issue. We faced a similar issue with DXA and the fix was to change the underlying DD4T serializer. You will have to lock a static object to avoid concurrency. Link to a base serialization service can be found here. Link to the actual transaction commit is here.

Hope it helps resolve the issue.

3
  • Thanks @Shiva for answer. About the code modification, should we open an issue on GitHub or we need to modify it by ourselves? Thank you:)
    – Ya Ping
    Jul 30, 2018 at 6:03
  • The fix for this is implemented with DD4T 2.2. github.com/dd4t/DD4T.Model/blob/2.2.2/source/DD4T.Serialization/… One option would be to manually modify it yourself or choose the tag with the implemented fix. There are a couple of closed issues already with DD4T for this but with different errors. You can update the closed defect with your error, if the fix works for you. Hope it helps!
    – Shiva
    Jul 30, 2018 at 15:54
  • Hello @Shiva: We modify DD4T.Model-2.2.1 manually and it works for us. Thanks your information so much:) We modify two files as below: DD4T.Model-2.2.1\source\DD4T.Serialization\XmlSerializerService.cs DD4T.Model-2.2.1\source\DD4T.Serialization\JSONSerializerService.cs
    – Ya Ping
    Nov 29, 2018 at 3:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.