I have the following:
- SDL Web 8.5 AWS cloud setup
- 1 deployer endpoint, dedicated instance
- 2 deployer workers, dedicated instances
- JMS notifications between deployers (Amazon SQS)
- Redis BinaryStore (Amazon ElastiCache)
- 2 Publishers, dedicated instances
The issue: While doing a publishing load test, the deployer workers stop momentarily processing the transport packages, then they resume. In the end everything is published and deployed correctly. However, why the pause?
There are roughly 5k items that are being published. The Publishers fill up the queue. The Deployer endpoint pushes all the transport packages into the Redis BinaryStore and send JMS notifications. The Deployer workers receive notifications and proceed to deploy the transport packages.
Roughly 12 minutes into the process (consistently after many test runs), the workers pause for about 7 minutes, then they resume successfully. There are no errors in the logs. All transport packages are successfully deployed. No errors anywhere. The Redis storage is far from being full. Why the pause?
During the pause the number of items "Waiting for Deployment" spikes to roughly 2k, but once the workers resume, they quickly deploy these packages successfully.
I tried with different parameters for number of workers, receive timeouts on JMS side, connectivity timeout on Redis adapter. No effect.
Has anybody seen this? Why is it happening? It's not really a bug, because functionally it eventually works, but performance is hindered, as the deploy job finishes quite some time after the rendering/transporting has been done by the Publishers.
