We have upgraded from Tridion 2013 SP1 + DD4T 1.0 to SDL Web 8.1.1 micro services + DD4T 2.0 + SDL Web8 CIL proivder
Summary of the issue - Initially, we did upgrade SDL Web 8.1.1 + DD4T 2.0 legacy method without micro services, we had a CPU spikes issue it was working fine everything as expected but some behaviour, for example, 100times or 1000 times working fine and then some trigger trying to execute broker query or taxonomy keyword meta query CPU goes high not coming down because of that website goes down. - The following action items were done to investigated took the crash dump file during the CPU high to find out the route cast in the IIS worker app pool in process java JVM objects takes High CPU also noticed private memory also increasing from ~2GB to ~6GB
- And then decided to upgraded to micro-services, and then CPU issue was gone but we are getting now new issue very often getting this below error message and website getting 500 Internal server error.
None of the database broker database queries was succedded, we also noticed broker databse CPU was normal at the time.
06:48:49.29 Error Handle Section At Error occured at HandleSectionErrorAttribute handling - Error occured at
Sdl.Web.Delivery.Service.AbstractODataService : HandleException and Handling (http://www.sharjah24.ae/ar/uae/209359-%D8%AF%D9%8A%D9%88%D8%A7%D9%86-%D8%AD%D8%A7%D9%83%D9%85-%D8%A7%D9%84%D9%81%D8%AC%D9%8A%D8%B1%D8%A9-%D9%8A%D9%86%D8%B9%D9%89-%D8%A7%D9%84%D9%85%D8%BA%D9%81%D9%88%D8%B1-%D9%84%D9%87-%D8%B3%D9%85%D9%88-%D8%A7%D9%84%D8%B4%D9%8A%D8%AE-%D8%AD%D9%85%D8%AF-%D8%A8%D9%86-%D8%B3%D9%8A%D9%81-%D8%A7%D9%84%D8%B4%D8%B1%D9%82%D9%8A by 162.158.78.40) ... Sdl.Web.Delivery.Service.ServiceQueryException (Sdl.Web.Delivery.Service) ... Failed to execute query ... at Sdl.Web.Delivery.Service.AbstractODataService.HandleException(Exception e) ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_0
1.<Execute>b__0() ... at Sdl.Web.Delivery.Service.Utils.PerformRetry[T](Func
1 block, Int32 retryCount) ... at Sdl.Web.Delivery.Service.Utils.Retry[T](Func1 block, Int32 retryCount, Boolean locking) ... at Sdl.Web.Delivery.Dynamic.ComponentPresentationFactory.ExecuteComponentPresentationMetaDAOMethod(Int32 publicationId, String methodName, Object[] parameters) ... at Sdl.Web.Delivery.Dynamic.ComponentPresentationFactory.GetComponentPresentationWithHighestPriority(Int32 componentId) ... at Indivirtual.SHARJAH24.Common.TridionComponentPresentationProvider.GetContent(String uri, String templateUri) ... at DD4T.Factories.ComponentPresentationFactory.TryGetComponentPresentation(IComponentPresentation& cp, String componentUri, String templateUri) ... at DD4T.Factories.ComponentFactory.TryGetComponent(String componentUri, IComponent& component, String templateUri) ... at Indivirtual.DD4T.Mvc.Controllers.BasePageController.GetModelForPage(String PageId, PageType type) ... at Indivirtual.SHARJAH24.Controllers.PageController.Page(String pageId) ... at lambda_method(Closure , ControllerBase , Object[] ) ... at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary
2 parameters) ... at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary2 parameters) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState) ... at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult
2.CallEndDelegate(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.b__3d() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass2b.b__1c() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.b__1e(IAsyncResult asyncResult) ... System.NullReferenceException (Microsoft.OData.Client) ... Object reference not set to an instance of an object. ... at Microsoft.OData.Client.ClientEdmModel.<>c__DisplayClass19.b__d(IEdmSchemaElement et) ... at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source, Func
2 predicate) ... at Microsoft.OData.Client.ClientEdmModel.GetOrCreateEdmTypeInternal(IEdmStructuredType edmBaseType, Type type, PropertyInfo[] keyProperties, Boolean isEntity, Nullable1 hasProperties) ... at Microsoft.OData.Client.ClientEdmModel.GetOrCreateEdmType(Type type) ... at Microsoft.OData.Client.DataServiceContext.IsSingletonType[TElement]() ... at Microsoft.OData.Client.DataServiceContext.Execute[TElement](Uri requestUri, String httpMethod, OperationParameter[] operationParameters) ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_1
1.b__1() ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_01.<Execute>b__0() ...
1.b__0() ... at Sdl.Web.Delivery.Service.Utils.PerformRetry[T](Func
06:48:49.30 Error Timestamp 02-Apr-2017 06:48:44 06:48:49.30 Error Machine SMC-WWW2 06:48:49.30 Error Useragent Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/) 06:48:49.30 Error Client 162.158.78.40 (162.158.78.40) 06:48:49.30 Error Referrer
06:48:50.07 Error {0}: Could not execute query ... Sdl.Web.Delivery.Service.ServiceQueryException (Sdl.Web.Delivery.Service) ... Failed to execute query ... at Sdl.Web.Delivery.Service.AbstractODataService.HandleException(Exception e) ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_01 block, Int32 retryCount) ... at Sdl.Web.Delivery.Service.Utils.Retry[T](Func
1 block, Int32 retryCount, Boolean locking) ... at Sdl.Web.Delivery.Dynamic.ComponentPresentationFactory.ExecuteComponentPresentationMetaDAOMethod(Int32 publicationId, String methodName, Object[] parameters) ... at Sdl.Web.Delivery.Dynamic.ComponentPresentationFactory.GetComponentPresentationWithHighestPriority(Int32 componentId) ... at Indivirtual.SHARJAH24.Common.TridionComponentPresentationProvider.GetContent(String uri, String templateUri) ... at DD4T.Factories.ComponentPresentationFactory.TryGetComponentPresentation(IComponentPresentation& cp, String componentUri, String templateUri) ... at DD4T.Factories.ComponentFactory.TryGetComponent(String componentUri, IComponent& component, String templateUri) ... at Indivirtual.DD4T.Tridion.Broker.BrokerQuery.<>c.b__22_2(String c) ... at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext() ... at System.Linq.Enumerable.WhereEnumerableIterator
1.MoveNext() ... at System.Collections.Generic.List1..ctor(IEnumerable
1 collection) ... at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) ... at Indivirtual.DD4T.Tridion.Broker.BrokerQuery.Execute(Int32 pageIndex, Int32 pageSize) ... System.NullReferenceException (Microsoft.OData.Client) ... Object reference not set to an instance of an object. ... at Microsoft.OData.Client.ClientEdmModel.<>c__DisplayClass19.<GetOrCreateEdmTypeInternal>b__d(IEdmSchemaElement et) ... at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable
1 source, Func2 predicate) ... at Microsoft.OData.Client.ClientEdmModel.GetOrCreateEdmTypeInternal(IEdmStructuredType edmBaseType, Type type, PropertyInfo[] keyProperties, Boolean isEntity, Nullable
1 hasProperties) ... at Microsoft.OData.Client.ClientEdmModel.GetOrCreateEdmType(Type type) ... at Microsoft.OData.Client.DataServiceContext.IsSingletonTypeTElement ... at Microsoft.OData.Client.DataServiceContext.Execute[TElement](Uri requestUri, String httpMethod, OperationParameter[] operationParameters) ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_11.<Execute>b__1() ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_0
1.b__0() ...
06:48:50.07 Error Handle Section At Error occured at HandleSectionErrorAttribute handling - Error occured at System.Linq.Enumerable : FirstOrDefault and Handling (http://www.sharjah24.ae/ar/ by 162.158.92.166) ... System.ArgumentNullException (System.Core) ... Value cannot be null. ... Parameter name: source ... at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source, Func
2 predicate) ... at Indivirtual.SHARJAH24.Model.ModelExtension.MainArticles(IPage page, Int32 mainArticleCount, Boolean populateAuthors) ... at Indivirtual.SHARJAH24.Controllers.PageController.GetIndexModel(IPage page) ... at Indivirtual.SHARJAH24.Controllers.PageController.GetView(IPage page) ... at Indivirtual.SHARJAH24.Controllers.PageController.Page(String pageId) ... at lambda_method(Closure , ControllerBase , Object[] ) ... at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary2 parameters) ... at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary
2 parameters) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState) ... at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult2.CallEndDelegate(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass2b.<BeginInvokeAction>b__1c() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult) ...
1.b__0() ... at Sdl.Web.Delivery.Service.Utils.PerformRetry[T](Func
06:48:50.07 Error Timestamp 02-Apr-2017 06:48:45 06:48:50.07 Error Machine SMC-WWW2 06:48:50.07 Error Useragent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0 06:48:50.07 Error Client 162.158.92.166 (162.158.92.166) 06:48:50.07 Error Referrer
06:48:57.05 Error Handle Section At Error occured at HandleSectionErrorAttribute handling - Error occured at Sdl.Web.Delivery.Service.AbstractODataService : HandleException and Handling (http://www.sharjah24.ae/ar/sports/international/198729- by 141.101.76.154) ... Sdl.Web.Delivery.Service.ServiceQueryException (Sdl.Web.Delivery.Service) ... Failed to execute query ... at Sdl.Web.Delivery.Service.AbstractODataService.HandleException(Exception e) ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_01 block, Int32 retryCount) ... at Sdl.Web.Delivery.Service.Utils.Retry[T](Func
1 block, Int32 retryCount, Boolean locking) ... at Sdl.Web.Delivery.Dynamic.ComponentPresentationFactory.ExecuteComponentPresentationMetaDAOMethod(Int32 publicationId, String methodName, Object[] parameters) ... at Sdl.Web.Delivery.Dynamic.ComponentPresentationFactory.GetComponentPresentationWithHighestPriority(Int32 componentId) ... at Indivirtual.SHARJAH24.Common.TridionComponentPresentationProvider.GetContent(String uri, String templateUri) ... at DD4T.Factories.ComponentPresentationFactory.TryGetComponentPresentation(IComponentPresentation& cp, String componentUri, String templateUri) ... at DD4T.Factories.ComponentFactory.TryGetComponent(String componentUri, IComponent& component, String templateUri) ... at Indivirtual.DD4T.Mvc.Controllers.BasePageController.GetModelForPage(String PageId, PageType type) ... at Indivirtual.SHARJAH24.Controllers.PageController.Page(String pageId) ... at lambda_method(Closure , ControllerBase , Object[] ) ... at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary2 parameters) ... at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary
2 parameters) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState) ... at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult2.CallEndDelegate(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass2b.<BeginInvokeAction>b__1c() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult) ... System.NullReferenceException (Microsoft.OData.Client) ... Object reference not set to an instance of an object. ... at Microsoft.OData.Client.ClientEdmModel.<>c__DisplayClass19.<GetOrCreateEdmTypeInternal>b__d(IEdmSchemaElement et) ... at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable
1 source, Func2 predicate) ... at Microsoft.OData.Client.ClientEdmModel.GetOrCreateEdmTypeInternal(IEdmStructuredType edmBaseType, Type type, PropertyInfo[] keyProperties, Boolean isEntity, Nullable
1 hasProperties) ... at Microsoft.OData.Client.ClientEdmModel.GetOrCreateEdmType(Type type) ... at Microsoft.OData.Client.DataServiceContext.IsSingletonTypeTElement ... at Microsoft.OData.Client.DataServiceContext.Execute[TElement](Uri requestUri, String httpMethod, OperationParameter[] operationParameters) ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_11.<Execute>b__1() ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_0
1.b__0() ...
06:48:57.05 Error Timestamp 02-Apr-2017 06:48:57 06:48:57.05 Error Machine SMC-WWW2 06:48:57.05 Error Useragent Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) 06:48:57.05 Error Client 141.101.76.154 (141.101.76.154) 06:48:57.05 Error Referrer
06:49:07.64 Error Handle Section At Error occured at HandleSectionErrorAttribute handling - Error occured at Sdl.Web.Delivery.Service.AbstractODataService : HandleException and Handling (http://www.sharjah24.ae/ar/sharjah/172266-%D8%A7%D9%84%D9%85%D9%86%D8%B5%D9%88%D8%B1%D9%8A-%D9%86%D8%B3%D8%AA%D9%82%D9%8A-%D8%A7%D9%84%D8%AA%D8%B3%D8%A7%D9%85%D8%AD-%D9%85%D9%86-%D8%AA%D8%B9%D8%A7%D9%84%D9%8A%D9%85-%D8%AF%D9%8A%D9%86%D9%86%D8%A7-%D9%88%D9%85%D9%86%D9%87%D8%AC-%D9%82%D9%8A%D8%A7%D8%AF%D8%AA%D9%86%D8%A7 by 108.162.245.250) ... Sdl.Web.Delivery.Service.ServiceQueryException (Sdl.Web.Delivery.Service) ... Failed to execute query ... at Sdl.Web.Delivery.Service.AbstractODataService.HandleException(Exception e) ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_01.<Execute>b__0() ... at Sdl.Web.Delivery.Service.Utils.PerformRetry[T](Func
1 block, Int32 retryCount) ... at Sdl.Web.Delivery.Service.Utils.Retry[T](Func1 block, Int32 retryCount, Boolean locking) ... at Sdl.Web.Delivery.Dynamic.ComponentPresentationFactory.ExecuteComponentPresentationMetaDAOMethod(Int32 publicationId, String methodName, Object[] parameters) ... at Sdl.Web.Delivery.Dynamic.ComponentPresentationFactory.GetComponentPresentationWithHighestPriority(Int32 componentId) ... at Indivirtual.SHARJAH24.Common.TridionComponentPresentationProvider.GetContent(String uri, String templateUri) ... at DD4T.Factories.ComponentPresentationFactory.TryGetComponentPresentation(IComponentPresentation& cp, String componentUri, String templateUri) ... at DD4T.Factories.ComponentFactory.TryGetComponent(String componentUri, IComponent& component, String templateUri) ... at Indivirtual.DD4T.Mvc.Controllers.BasePageController.GetModelForPage(String PageId, PageType type) ... at Indivirtual.SHARJAH24.Controllers.PageController.Page(String pageId) ... at lambda_method(Closure , ControllerBase , Object[] ) ... at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary
2 parameters) ... at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary2 parameters) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState) ... at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult
2.CallEndDelegate(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.b__3d() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.b__3f() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass2b.b__1c() ... at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.b__1e(IAsyncResult asyncResult) ... System.NullReferenceException (Microsoft.OData.Client) ... Object reference not set to an instance of an object. ... at Microsoft.OData.Client.ClientEdmModel.<>c__DisplayClass19.b__d(IEdmSchemaElement et) ... at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source, Func
2 predicate) ... at Microsoft.OData.Client.ClientEdmModel.GetOrCreateEdmTypeInternal(IEdmStructuredType edmBaseType, Type type, PropertyInfo[] keyProperties, Boolean isEntity, Nullable1 hasProperties) ... at Microsoft.OData.Client.ClientEdmModel.GetOrCreateEdmType(Type type) ... at Microsoft.OData.Client.DataServiceContext.IsSingletonType[TElement]() ... at Microsoft.OData.Client.DataServiceContext.Execute[TElement](Uri requestUri, String httpMethod, OperationParameter[] operationParameters) ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_1
1.b__1() ... at Sdl.Web.Delivery.Service.ODataV4Service.<>c__DisplayClass15_0`1.b__0() ...
Anyone based this issue, Any help would be highly appreciated.