1

Beginning in early July, developers see that bulk publishing has a high rate of failure for a Web 8.5 UAT environment with the si4t Deployer extension implemented. Apprx. 30 pages with 12 components each can be published successfully without failure, but when more pages are published at once, frequent failures are seen. For example, when publishing 170 pages, each with 12 components, the first 20 pages were successfully published and the subsequent 150 pages failed. The main error seen in the debug deployer log at time of error is as below; occasionally there are other errors seen but this is the only one which is seen consistently.

2020-09-02 08:32:41,262 DEBUG CommandQueuePoller - No more processing slots available on queue: ContentQueue
2020-09-02 08:32:42,278 DEBUG CommandQueuePoller - No more processing slots available on queue: ContentQueue
2020-09-02 08:32:43,278 DEBUG CommandQueuePoller - No more processing slots available on queue: ContentQueue
2020-09-02 08:32:44,278 DEBUG CommandQueuePoller - No more processing slots available on queue: ContentQueue
2020-09-02 08:32:44,887 ERROR TridionCommitStep - Commit failed for execution id:tcm:0-569010-66560
com.sdl.delivery.deployer.api.processing.exception.ProcessingException: Unable to commit transactions. ExecutionId:tcm:0-569010-66560
    at com.sdl.delivery.deployer.steps.TridionTransactionalExecutableStep.commitTransaction(TridionTransactionalExecutableStep.java:78)
    at com.sdl.delivery.deployer.steps.TridionCommitStep.process(TridionCommitStep.java:28)
    at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.execute(StepActor.scala:72)
    at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.com$sdl$delivery$deployer$engine$pipeline$actor$StepActor$processStep(StepActor.scala:57)
    at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor$$anonfun$receive$1.applyOrElse(StepActor.scala:40)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
    at com.sdl.delivery.deployer.engine.pipeline.actor.StepActor.aroundReceive(StepActor.scala:26)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.tridion.broker.StorageException: Commit failed for transaction tcm:0-569010-66560 because of Server at http://HOSTNAME:PORT/solr/loremipsum returned non ok status:502, message:Proxy Error
    at com.tridion.storage.management.TransactionManagerImpl.commitTransaction(TransactionManagerImpl.java:70)
    at com.tridion.storage.StorageManagerFactory.commitTransaction(StorageManagerFactory.java:226)
    at com.sdl.delivery.deployer.steps.TridionTransactionalExecutableStep.commitTransaction(TridionTransactionalExecutableStep.java:76)

Environment details:

  • Deployer pool configuration
<Pool Type="jdbc" Size="50" MonitorInterval="60" IdleTimeout="120"  
CheckoutTimeout="120" MinEvictableIdleTimeout="-1" ValidationQuery="SELECT 1"  
TestOnBorrow="true"/>
  • Broker database optimized
ALTER DATABASE <Tridion_Broker> SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE <Tridion_Broker> SET READ_COMMITTED_SNAPSHOT ON
EXEC SP_UPDATESTATS
  • State store constraints added, as discussed in KB article
  • The UAT environment only is encountering issue. Issue is not seen on the Production environment.
  • In both UAT and Production, there are two load balanced Deployer servers, and two Solr servers.
  • Solr 7.4 is installed in all environments, w/64-bit Java 1.8.0_181. JVM args are identical across environments. 16 GB physical memory allocated, 25% swap space, JVM memory usage is minimal at less than 5%. Disk space adequate on both environments. In the Deployer lib directory, the si4t jars used are si4t-solr-1.2.jar, si4t-se-1.3.jar, solr-solrj-4.4.0.jar, and solr-core-4.4.0.jar. The 6 extra dependency jars for Web 8 are also resident in the Deployer
  • There are about 225 URLs in the Deployer Indexer class.

Troubleshooting done:

  • When the si4t extension is removed, apprx 30 pages can be published successfully without error. 200 pages can be published successfully with 4 errors, but these errors look to be related to rendering and workflow so not related to the Deployer.
  • Comparison done of the Production and UAT Solr environments, no differences in specs or configuration found.
  • Solr logging reviewed, no helpful info found.
  • Issue persists immediately after restarting Deployer services and Solr.
  • After reproducing failures with bulk publishing, publishing singular or a small number of pages for the failed publish transactions is successful.
  • Review of "netstat -all" output from Deployer and Solr servers at the time of failures.

It seems that there is some threshold in SOLR being hit when bulk publishing, but it's not clear how Solr or the Deployer could be tuned to allow a higher throughput.

Any feedback would be much appreciated.

Update :

There were no errors on the Solr side with logging level raised to ALL, where 58 items were published and only the first 22 were successfully published, and the remainder failed .

  • Hi Sayoni, It seems SI4T is unable to connect the SOLR or maybe that URL is not reachable. Could you try to hit that SOLR URL from your deployer server and check the response? – Sayantan Basu Aug 31 at 20:37
  • Hi @Sayoni, is it possible for you to see what's happening in the Solr logs? – Raimond Sep 1 at 7:04
  • @Raimond For the solr logging, at error level only the below error is seen which seems unrelated, but I will raising the log level and will update forum post with the more verbose logs. Commons Daemon procrun stderr initialized ERROR StatusLogger Unable to access file:/F:/Solr/apache-solr/server/file:F:/Solr/apache-solr/server/scripts/cloud-scripts/log4j2.xml java.io.FileNotFoundException: F:\Solr\apache-solr\server\file:F:\Solr\apache-solr\server\scripts\cloud-scripts\log4j2.xml – Sayoni Sep 2 at 8:13
  • Hi @SayantanBasu ,even when the publishing fails , Solr admin UI appears for the indexing url when put in browser from the deployer server . – Sayoni Sep 2 at 13:54
  • @Sayoni ok, please do. You don't have a proxy running in between right? – Raimond Sep 3 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.