I have a AWS scaled-out Content Delivery setup with 1 Deployer endpoint and 2 Deployer workers, each on their own server (so 3 servers in total).
They all have r/w access to a shared files system (EFS on AWS).
The Deployers work fine, but it seems the performance is low.
Also I see numerous errors like the following in the core and deployer logs on each Deployer Worker servers:
ERROR FSDeployerQueue - Queue [Id=ContentQueue]. Failed to delete serialized
command '/efs01/deployer-queues/ContentQueue/tcm_0-106038-66560.Content.json'
All transactions are successful, but nevertheless there are hundreds of these errors in the logs. It seems the .json files are locked and they can't be deleted. These files stay behind for days.
The workers seem to be busy attempting to delete these files for minutes at a time and that draws performance down.
I played with different number of workers threads (5, 10, 20, 30) but the load on the CPU there is no noticeable difference. It bursts to 70-90% for a few seconds, then it lingers below 5% for minutes.
I have installed cummulative hotfix CD_8.5.0.6340 with the latest Deployer fixes. This has no effect on the issue.
I have played with different settings for the location polling interval (2s, 5s, 10s), but there is no significant impact on the issue.
My queue settings in deployer-conf.xml
look the following:
<Queue Default="true" Verbs="Content" Adapter="FileSystem" Id="ContentQueue">
<Property Name="Destination" Value="/efs01/deployer-queues"/>
<Property Name="Workers" Value="30"/>
</Queue>
<Queue Verbs="Commit,Rollback" Adapter="FileSystem" Id="CommitQueue">
<Property Name="Destination" Value="/efs01/deployer-queues"/>
<Property Name="Workers" Value="30"/>
</Queue>
<Queue Verbs="Prepare" Adapter="FileSystem" Id="PrepareQueue">
<Property Name="Destination" Value="/efs01/deployer-queues"/>
<Property Name="Workers" Value="30"/>
</Queue>
<Adapter Id="FileSystem">
<Property Name="LocationPollingInterval" Value="10s"/>
</Adapter>
Have you seen this error before? What is the fix? Is it a bug? Am I doing some crazy weird setup? Thanks!