Fredhopper to Elastic Search Migration "Failed to Query Elastic Search"

Question

We are currently in the process of upgrading from Fredhopper to Elastic Search on SDL Web 9.1. The migration from Fredhopper to Elastic Search appears to have gone smoothly but we have now run into an issue with our CMS’s Targeting dashboard and the xo management micro-service. When visiting the Targeting Dashboard we don’t see any promotions loading, and we see errors in the notification center.

The following are details for our new environment:

• xo management version: 11.1.0-xxxx - Installed on Linux (Java 11)
• SDL Web 9.1 (Installed on Windows Server 2019)
• Elastic Search 6.7.0 – Installed on Linux

Steps to reproduce:

1. We have a very large number of promotions created in the Targeting CMS interface (over 150). We have over 12 regions and over 20 triggers.
2. We have hundreds of promotions published using the SmartTarget TBBs.
3. When we login to the CMS and attempt to hit the Targeting screen we see the following:

Checking the details we can see the following error:

at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, QueryComponents queryComponents)
   at System.Data.Services.Client.DataServiceQuery`1.GetEnumerator()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Tridion.Web.UI.Models.SmartTarget.ContentDelivery.ODataServiceClient.get_LiveAttributes()
   at Tridion.Web.UI.Models.SmartTarget.LiveAttributes.GetLiveAttributes(String instanceId)
   at Tridion.Web.UI.Models.SmartTarget.ExternalDataProcessor.AddExternalData(IEnumerable`1 promotions, String instanceId)
   at Tridion.Web.UI.Models.SmartTarget.Promotions.GetPromotions(Filter filter)
   at Tridion.Web.UI.Models.SmartTarget.Services.GetPromotions(Filter filter)
   at SyncInvokeGetPromotions(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at Tridion.Web.UI.Core.Extensibility.DataExtenderOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

Checking the JavaScript Console on the CMS during page load, we can see the following error:

POST http://url/WebUI/Models/SmartTarget/Services.svc/GetPromotions Failed to load resource: the server responded with a status of 500 (Internal Server Error) 

I would also note that we get the same issue if we hit the xomanagement service directly at the following url: http://url/management.svc/Attributes/

Finally, I would also note that the xo management service returns correctly if we hit any other URL, including: http://url/management.svc/TriggerTypes and http://url/management.svc/Promotions

Is there any way to get around this issue for a large number of promotions. We have seen this post where it is possible to configure the max buffer size as part of the request: https://stackoverflow.com/a/51020999/5040955 Is there any way to do that for the XO Management Queries?