Usage of EHCache and Error while registering EHCache caused outage

Question

We are using DXA 1.7 and the spring MVC project contains a file with eh-cache settings. Can someone please let me know what is the purpose of this cache and what is it caching and what are the optimal settings for the caching to be used in production. This is how our production ehcache looks like

<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd" updateCheck="false">


    <defaultCache
            eternal="false"
            maxEntriesLocalHeap="10000"
            timeToLiveSeconds="60"
            timeToIdleSeconds="60"
            memoryStoreEvictionPolicy="LRU">
    </defaultCache>
    <cache name="DD4T-Objects"
           eternal="false"
           maxEntriesLocalHeap="10000"
           timeToLiveSeconds="60"
           timeToIdleSeconds="60"
           memoryStoreEvictionPolicy="LRU">
    </cache>
    <cache name="DD4T-Dependencies"
           maxEntriesLocalHeap="50000"
           memoryStoreEvictionPolicy="LRU">
    </cache>
</ehcache>

The reason I am interested in knowing this because our site crashed on live and the logs are only filled with

 org.ehcache.jsr107.Eh107CacheManager - Registering Ehcache MBean javax.cache:type=CacheStatistics,CacheManager=urn.X-ehcache.jsr107-default-config,Cache=com.sdl.web.discovery.datalayer.model.ContextServiceCapability

The server tried to register EHCache for around 12 hours until the machine ran out of heap space and died. Interestingly before this error started appearing the Tridion service was down for 5 minutes

Error in com.belmond.travelstudio.controller.searchPackages.handleGetPage! com.sdl.odata.client.api.exception.ODataClientHttpError: Unable to get response from OData service: No Response.

I am trying to connect the possible dots here, I think the OData service went down and EHCache became empty and then it never woke up.If that was the cause should not it be handled by dxa web app ? If you think these issues are unrelated please feel free to comment.

We have 3 instances of DXA running and all of them went down one after another at same time.

Answer 1

In a DXA 1.7 Web App, EHCache is used by both DD4T and CIL. The log messages reveal that the issue is in a section of the CIL cache which caches Discovery Service Capabilities.

Your theory that the issue is induced by unavailability of the Context Service seems plausible. If this is reproducible, it should be reported with SDL Customer Support as a potential CIL bug.