We are experiencing page load times between 5-15 seconds (sometimes more) when applying a small load of approximately 40 users to our web application. This is for a simple page with just one static <div> on it. Investigation uncovered that the Content service is consuming approximately 50% of our server CPU which maxes out the CPU and hence leads to the long page loads. 

Removing the “SDL Ambient Client Filter” from our web.xml reduces page load to within 200 milliseconds, and reduces the Content service CPU to just 1% (even with 100+ active users). 

We have pin-pointed the issue to the SDL Ambient Client Filter which is making two calls to the Content service for every page load:
/client/v4/content.svc/RequestEndActionImport
/client/v4/content.svc/RequestStartActionImport

The offending code is in the class “com.sdl.web.ambient.client.AmbientClientFilter” inside the “content-ambient-client-*.jar”.  Specifically the following lines:

    Builder actionBuilder = new Builder().withReturnType(String.class).withActionName(isRequestStart ? "RequestStartActionImport" : "RequestEndActionImport").withActionParameter("ClaimStore", String.format("\"%s\"", new Object[]{SERIALIZATION_UTIL.serializeClaimStore(claimStore)}));
    
    	. . .
    
    String claimStoreStr = StringEscapeUtils.unescapeHtml4((String) ContentClientProvider.getInstance().getContentClient().performAction(actionBuilder.build(), false));


All other calls to the Content service are cached on the webapp level (as configured in the cd_client_conf.xml), hence the huge performance improvement without the ADF enabled.

When we looked into the calls above, and we found that they are returning some claim values, and claim scopes (claims like useragent:browser, os:model, tracking:id, and session:id).  We're not sure if this is necessary or if there is some way to avoid this large overhead when using the ADF.
 

Here is the filter as configured in our Web.xml:

        <filter>
    		<filter-name>SDL Ambient Client Filter</filter-name>
    		<filter-class>com.sdl.web.ambient.client.AmbientClientFilter</filter-class>
    	</filter>
    	<filter-mapping>
    	   <filter-name>SDL Ambient Client Filter</filter-name>
    	   <url-pattern>/*</url-pattern>
    	</filter-mapping>

Here are some useful implementation details:

    Language = Java   
    CM = SDL Web 8.5  
    WebApp CIL = com.sdl.web -> cil -> 8.5.0-1056  
    Microservices = Hotfix CD_8.5.0.15928  
    Webserver = WebSphere Custom
    Implementation = CIL: Not DXA, Not DD4T
    Add ons: SmartTarget with custom cartridges

Any ideas on why these calls are occurring for every page request or if there is some way to cache/disable them?

Edit: Note that we are using SmartTarget actively on some pages.