4

I am getting a deadlock error in the deployer log at the time of publishing to staging environment. Though there is no issue while publisshing to Live. The error is not consistant but due to this the staging publishing job stucks and finally fails. Here is the error Log for the deployer.

    2017-10-18 08:31:21,338 ERROR SqlExceptionHelper - Transaction (Process ID 126) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
    2017-10-18 08:31:21,339 ERROR ExceptionMapperStandardImpl - HHH000346: Error during managed flush [org.hibernate.exception.LockAcquisitionException: could not execute statement]
    2017-10-18 08:31:21,341 ERROR [dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute statement] with root cause
    com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 126) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
        at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1655)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:440)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:385)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:328)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
        at org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45)
        at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3246)
        at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3483)
        at org.hibernate.action.internal.EntityDeleteAction.execute(EntityDeleteAction.java:99)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:582)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:456)
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
        at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1407)
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:477)
        at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3170)
        at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2384)
        at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:467)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:146)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$100(JdbcResourceLocalTransactionCoordinatorImpl.java:38)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:220)
        at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:68)
        at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:517)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:487)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy124.deleteExecution(Unknown Source)
        at com.sdl.delivery.deployer.legacy.endpoint.controller.DeployerLegacyEndpointController.removeStateFile(DeployerLegacyEndpointController.java:236)
        at com.sdl.delivery.deployer.legacy.endpoint.controller.DeployerLegacyEndpointController.doGet(DeployerLegacyEndpointController.java:195)
        at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at com.tridion.versioning.VersionFilter.doFilter(VersionFilter.java:46)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at com.sdl.web.readwrite.filter.ReadWriteFilter.doFilter(ReadWriteFilter.java:57)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at com.tridion.ambientdata.web.AbstractAmbientDataServletFilter.doFilter(AbstractAmbientDataServletFilter.java:286)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
        at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1697)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:934)
        at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$4.completed(Nio2Endpoint.java:626)
        at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$4.completed(Nio2Endpoint.java:604)
        at sun.nio.ch.Invoker.invokeUnchecked(Unknown Source)
        at sun.nio.ch.Invoker$2.run(Unknown Source)
        at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)

Also I am getting another error when the job stuck in the publishing queue for a long time. 

    2017-10-18 09:25:16,259 ERROR DefaultOAuthToken - Token expired.
    2017-10-18 09:25:16,263 ERROR DefaultOAuthToken - Token expired.
    2017-10-18 09:25:16,266 ERROR DefaultOAuthToken - Token expired.
    2017-10-18 09:25:16,268 ERROR DefaultOAuthToken - Token expired.

I have set the same configuration for Live and Staging in the deployer configuration. Below is the Publisher settings enter image description here

And set the Workers value as 10 in the deployer configuration. enter image description here

Do I need to change any configuration to avoid the deadlock?

I am using sdl Web 8.5.

Improve this question
3
  • Are you using same configuration for live? Are you performing bulk publish or phased publish?
    – Hiren Kaku
    Commented Oct 18, 2017 at 22:47
  • @Hiren : It is the same. Except the live discovery service is configured with 2 deployers using mirror strategy.
    – Manas.p
    Commented Oct 19, 2017 at 2:22
  • A deadlock is a database issue, so that is the system you should focus on. What could be important here is the exact setup of the deployer (or multiple deployers) and also if the database is properly maintained. I suggest you edit your question and supply a few more relevant details around all this.
    – Bart Koopman
    Commented Oct 19, 2017 at 8:10

2 Answers 2

Reset to default
2

One more question,

the pages / components that you are trying to publish have common sub-components?

We had faced similar issue when sub-components were same then one transaction will try to delete and re-insert the sub-component and other transaction also trying to do the same for the same sub-component and it was creating deadlock.

In such a scenario, two things helped:

  1. Performing the database tuning that helped with overall performance however did not help with the actual issue.
  2. For the actual issue since we were doing bulk publish, we added delay of few seconds between multiple publishes, (in other words phased publishing). This helped to resolve the issue.

Please note, we had faced issue with Tridion 2013 sp1 hr2 and you are on web 8.5 so I would strongly recommend to open support ticket with the stack trace so that support team can validate and confirm on the same.

Improve this answer
2
  • Hi Hiren, Can you suggest how did you handle the phased publishing?
    – Manas.p
    Commented Oct 20, 2017 at 16:18
  • As suggested, we were performing bulk publishing so in the tool used for bulk publishing we added delay of few seconds after few transactions. For the normal publishing you can use in built phased publishing approach where you can say render and publish later and don't schedule all the publish later to the same time. Hope this helps.
    – Hiren Kaku
    Commented Oct 20, 2017 at 17:44
0

For now I removed the zip files from the service folder and set the clean up parameter to true. So the service folder looks to be clean now and after this I have not found any deadlock error in the deployer log. I am monitoring the issue and will update the question if I find any error related to this.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.