1

We moved our websites to production environment and have been observing random slow response from random pages on all websites. Whenever we test it’s a different page that is slow to respond. It does happen randomly but mostly happens with dynamic pages.

Configuration:

  • 2 load balanced web servers with 14gb RAM

  • DXA version 2.0.1

  • 2 load balanced servers with 64GB RAM for microservices
  • 72hrs caching on model service and 8GB memory for object caching. 6GB for content service caching
  • Output caching enabled on web application for 30min

We have not been able to replicate the issue in any other environment. We even tried pointing our local environment to production microservices and were unable to replicate the issue.

One difference between staging/production is the output-caching duration which is 30min for production (we assumed better performance with that) but its 2min in staging.

As per our infra team, there have not been any high memory or cpu usage. Our db has sp_updatestats enabled and as I mentioned, if we point our local environment to production db, then the issue doesnt occur. All dynamic pages load within 4 sec

Our thought is with the web servers only and may be, because there is more traffic on it than staging, may be IIS is unable to handle with 30min caching. But we are not sure how to confirm our suspicions or what else can it be.

Any suggestions that might help us would be greatly appreciated!

Improve this question
5
  • Please provide bit more details, of your implementations caching layers and settings of content service, model service and web app servers, and if any CDN used? what is the model service version used? Is it inprocess model service or CIL based model service?, during the random slow responses Is there any traffic peak in content service or model service? update to share the model service and content service JVM memory and object caching LRU value?
    – Velmurugan ♦
    Commented Sep 22, 2019 at 11:24
  • Could you also check IIS is not randomly recycling it's app pool? This would also cause slowness. Do you have an idea of the amount of traffic you are seeing in production? Maybe you could try to simulate this traffic in staging to see if you can replicate it there. There are several tools that allow that (for example have a search for jmeter).
    – Harald Hoffelinck
    Commented Sep 22, 2019 at 19:34
  • @Velmurugan: As mentioned in the question, we are using DXA 2.0.1 which directly connects with Broker DB and the cache settings are also mentioned above. We are caching everything for 72hrs with ActiveMQ. There is actually very minimal traffic on our production servers as of now, just a few hundred users. There are no high memory peaks on any of the servers as per the information from our server team
    – GJin
    Commented Sep 24, 2019 at 18:05
  • @HaraldHoffelinck: We checked that as well and our server team is using all standard/default IIS settings. The only other thing that we thought was that 14GB might not be enough for the websites but as per the infra team, there have not been any high memory or CPU usages till now.
    – GJin
    Commented Sep 24, 2019 at 18:06
  • Are you seeing any hanging threads in IIS? Did you have your networking team take a look at the load balancers as well? If it is the former, you are probably having memory leaks somewhere in your code and some form of log in your application or event system
    – Shiva
    Commented Sep 25, 2019 at 18:18

1 Answer 1

Reset to default
1

We were actually able to resolve the issues by creating custom indexes on the broker database. We ran query tests to see what queries were taking longer and added the recommended indexes. That improved the performance from 50sec to 3sec.

Improve this answer
1
  • Good that you found a solution. Still, I'd have thought that broker caching would take care of this. Maybe it's interesting to give that some attention.
    – Dominic Cronin
    Commented Oct 7, 2019 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.