Beginning in early July, developers see that bulk publishing has a high rate of failure for a Web 8.5 UAT environment with the si4t Deployer extension implemented. Apprx. 30 pages with 12 components each can be published successfully without failure, but when more pages are published at once, frequent failures are seen. For example, when publishing 170 pages, each with 12 components, the first 20 pages were successfully published and the subsequent 150 pages failed. The main error seen in the debug deployer log at time of error is as below; occasionally there are other errors seen but this is the only one which is seen consistently.
2020-09-02 08:32:41,262 DEBUG CommandQueuePoller - No more processing slots available on queue: ContentQueue
2020-09-02 08:32:42,278 DEBUG CommandQueuePoller - No more processing slots available on queue: ContentQueue
2020-09-02 08:32:43,278 DEBUG CommandQueuePoller - No more processing slots available on queue: ContentQueue
2020-09-02 08:32:44,278 DEBUG CommandQueuePoller - No more processing slots available on queue: ContentQueue
2020-09-02 08:32:44,887 ERROR TridionCommitStep - Commit failed for execution id:tcm:0-569010-66560
com.sdl.delivery.deployer.api.processing.exception.ProcessingException: Unable to commit transactions. ExecutionId:tcm:0-569010-66560
at com.sdl.delivery.deployer.steps.TridionTransactionalExecutableStep.commitTransaction(TridionTransactionalExecutableStep.java:78)
at com.sdl.delivery.deployer.steps.TridionCommitStep.process(TridionCommitStep.java:28)
at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.execute(StepActor.scala:72)
at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.com$sdl$delivery$deployer$engine$pipeline$actor$StepActor$processStep(StepActor.scala:57)
at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor$$anonfun$receive$1.applyOrElse(StepActor.scala:40)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.aroundReceive(StepActor.scala:26)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.tridion.broker.StorageException: Commit failed for transaction tcm:0-569010-66560 because of Server at http://HOSTNAME:PORT/solr/loremipsum returned non ok status:502, message:Proxy Error
at com.tridion.storage.management.TransactionManagerImpl.commitTransaction(TransactionManagerImpl.java:70)
at com.tridion.storage.StorageManagerFactory.commitTransaction(StorageManagerFactory.java:226)
at com.sdl.delivery.deployer.steps.TridionTransactionalExecutableStep.commitTransaction(TridionTransactionalExecutableStep.java:76)
Environment details:
- Web 8.5 (SQL Server), Deployer hotfix CD_8.5.0.18753 has been applied.
- Deployer has been configured with these steps as advised by SDL Support https://gateway.sdl.com/apex/communityknowledge?articleName=000011841
- Deployer pool configuration
<Pool Type="jdbc" Size="50" MonitorInterval="60" IdleTimeout="120"
CheckoutTimeout="120" MinEvictableIdleTimeout="-1" ValidationQuery="SELECT 1"
TestOnBorrow="true"/>
- Broker database optimized
ALTER DATABASE <Tridion_Broker> SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE <Tridion_Broker> SET READ_COMMITTED_SNAPSHOT ON
EXEC SP_UPDATESTATS
- State store constraints added, as discussed in KB article
- The UAT environment only is encountering issue. Issue is not seen on the Production environment.
- In both UAT and Production, there are two load balanced Deployer servers, and two Solr servers.
- Solr 7.4 is installed in all environments, w/64-bit Java 1.8.0_181. JVM args are identical across environments. 16 GB physical memory allocated, 25% swap space, JVM memory usage is minimal at less than 5%. Disk space adequate on both environments. In the Deployer lib directory, the si4t jars used are si4t-solr-1.2.jar, si4t-se-1.3.jar, solr-solrj-4.4.0.jar, and solr-core-4.4.0.jar. The 6 extra dependency jars for Web 8 are also resident in the Deployer
- There are about 225 URLs in the Deployer Indexer class.
Troubleshooting done:
- When the si4t extension is removed, apprx 30 pages can be published successfully without error. 200 pages can be published successfully with 4 errors, but these errors look to be related to rendering and workflow so not related to the Deployer.
- Comparison done of the Production and UAT Solr environments, no differences in specs or configuration found.
- Solr logging reviewed, no helpful info found.
- Issue persists immediately after restarting Deployer services and Solr.
- After reproducing failures with bulk publishing, publishing singular or a small number of pages for the failed publish transactions is successful.
- Review of "netstat -all" output from Deployer and Solr servers at the time of failures.
It seems that there is some threshold in SOLR being hit when bulk publishing, but it's not clear how Solr or the Deployer could be tuned to allow a higher throughput.
Any feedback would be much appreciated.
Update :
There were no errors on the Solr side with logging level raised to ALL, where 58 items were published and only the first 22 were successfully published, and the remainder failed .
