Broker Service not working as expected and Slow response of website

Question

I am working with legacy implementation in R5.3 SP1 and there are more than 100 websites most of them have dynamic content serve through the Broker API. We have a 3 Server Load Balancing set up - say Server 1 , Server 2 and Server 3. Here Server 1 is primary and publishing happens only on it and Server 2 and 3 are replicated from it using DFS. My concern is that almost daily Server 2 and 3 stops responding to web request while Server 1 is responding quickly.

I check the Server 2 and 3 for any logs, errors, utilization etc. and they all seems fine except the CPU usage of cd_broker. If we re-start this, the issue got resolved but even in that case the responses are significantly slower than Server 1.

I am referring THIS questions and suggestions provided here and will follow them, but the real question is that why only server 2 and 3 are creating problem and not server 1 since they all are using a centralized cluster of Broker DB?

Few finding:

• Lots of custom code in the form of batch jobs, broker API job etc. is there
• I have been told that DB maintenance, Tridion Re-Indexing and other maintenance related tasks are done periodically
• Nothing important is in the Event Logs

Can someone share his experience or knowledge what else should be checked.

Answer 1

There are quite a few things that can be checked. Its clear that there is a difference in the servers themselves or the way they are operating. Assuming they are just the same specs then I would look at:

1. The deployed application; are all the files the right versions
2. The configuration; caching on in all places? Is the config the same?
3. Is the network on the servers configured the same? Auto vs 1GB
4. Is there any differences in the load the servers are taking, load balancer issues
5. Assume that all the servers have resources left (memory, cpu)?

Answer 2

I would be suspicious of DFS. If it is copying the files too often, you will trigger unnecessary recompiles, and therefore application restarts.

You can investigate this by using perfmon and adding some ASP.NET counters, specifically CompilationsTotal and ApplicationRestarts. Then just check if the results correlate with your performance dropouts.

Answer 3

Thanks to both- Dominic and Julian for giving various suggestions based on which finally we are able to settle this issue. There were quite a few issues; first as per suggestion from Dom and Jules, we have used the windows performance monitor to figure out connection to IIS and identified that the load is not balanced evenly on all servers, also the server which was responding very well seems to be out of LB as not receiving any hit on it at all.

Apart from it, we have also noticed that another issue was caused by the DFS as along with it a Full Backup task also get executed suppressing the performance on the primary server and on all others.

Thanks again Dom and Jules and I will throw a dice to identify whose answer should I accept :) as both these answers helps me tremendously.