/WebUI/Models/TCM54/Services/General.svc/GetUserSettings taking TEN minutes to respond

Question

SDL Tridion 2011 SP1 HR1.

When I'm hitting the CME in the browser I see the GetUserSettings is stalling. It is taking 10 minutes - seriously... 10 minutes to receive a response. As detailed below, other TCM54 calls can be made following this and they are almost instant - resending the GetUserSettings request via Firebug takes 10 minutes again...

Watching the requests/SQL I note the following

The request on the live server is almost instant. On UAT (on the right hand side) the request is stalled waiting for a response… I can see (through SQL Trace) that the following query is sent to the CMS and returns the correct data almost immediately on hitting the URL:

But the browser stalls calling the webservice and, as shown below responds after a lengthy period of time – nearly TEN minutes every time +/- 30 seconds…

On following this through with Firebug I can see that the xmlHttp request is where the delay starts… I thought the JSON object being empty was significant this is the case in the working Live and problematic UAT instance.

So eventually the CMS hits up… BUT… At this point on testing more, I can confirm that we’re unable open/amend Pages whilst I can still create and amend Components.

Pages are checked out and then stall in the following state (when opening an existing Page):

I’m not seeing any errors/waiting calls in Firebug etc.

Oddly enough I can see other calls (GetLists etc) to the TCM54 Services are responding with a few ms as one would hope (these use the same xmlHttp request as above). This made me a little nervous about going straight in at the Network guys...

Half way through this investigation I became aware of an error from the Transport Logs:

DEBUG RollingStateRetriever - Unable to read transport state file, not adding to summary batch

So I tested and can see that I am also unable to publish. I’ve confirmed the license and checked all services (rebooted the machine) and have also confirmed the cd_storage, deployer etc look OK (and as shown above, the SQL DB is being queried) – the error that eventually returns is:

The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:10:00'.

I’ve raised the issue with the client but they confirm no network changes have been implemented over the weekend - is there any way I can test-out/confirm this.

I’ve confirmed the COM+ services appear to be in order…

So it's inconsistent in which calls, but consistent in the GetUserSettings call is always 10 minutes - and it's not clear why GetList and opening Components is OK whilst opening a Page fails .. or rather doesn't register a fail but simply doesn't complete.

Any pointers to an appropriate testing strategy to help nail what element(s) in the chain could be causing/contributing to this issue would be great.

Thanks

UPDATE: I tried the technique of disabling the Loopback Check (adding the registry DWORD DisableLoopbackCheck = 1 and rebooting) to no avail.

JAVA: (same version as live)

• java version "1.6.0_20"
• Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
• Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

The same hotfixes are applied per the working LIVE version.

Answer 1

It seems the only answer we could assume was

an internal network update relating to internal DNS settings.

Unfortunately this is all the information we really have at the moment although we'll push to investigate what was changed (if anything, from what... to what...) as it's very interesting that the xmlHttp response would take 10 minutes every time and not for all TCM54 webservice requests...

If anyone else comes across this issue and can shed more light than the response above I'd be most appreciative ... but at least you should have a little more confidence going in at the network guys :)

We did try local testing such as tracert etc. but we could see normal HTTP requests resolving fine upto the point of the webservice call to GetUserSettings. As I say above, the GetList (to list SGs/Pages) call to the same webservice resolved in a few milleseconds - every time.

UPDATE: PROBABLE CAUSE: Something we'd discarded as we didn't see a pattern that would implicate this was that we have an external application that imports content into the CMS based on exports from and external system. This application executes every hour but the error we report only occurred once or twice a day and not at the same time.

On further investigation we did find a pattern over the days that the error was always occurring around the hour.

Investigation of client network activities etc. turned up nothing then the realisation that this could be related to our app!

app background Within the app we upload content using the Core Service and for binary items we create a new socket an push the binary item into the CMS. Something we should have realised from the outset was that the client isn't a robot and the content is updated as and when required - not consistently in the sense of every hour would be the same!

Alas... when the client DID release a batch with a number of binary updated, the application was simply iterating through binary items and pushing them in using up all available sockets -

thus, it seems, the CMS was unable to connect over the network... I'm not sure why this would only affect certain webservice calls unless these are using different communication methods in the back end (something it will be nice to investigate when time permits) - but now we've updated the app to utilise a single socket we no longer see the same issue.

Answer 2

Not all requests are made from the browser to the webserver; for some requests the webserver connects back to itself. I would presume that in your case these are the "problematic" calls as apparently the browser could resolve the DNS name of the server without any problems. When the server->server connections take place in a blocking manner (probably going through some lengthy timeout) this will also block the AJAX request from the client.

You can see this in the IIS logs (the connection would be made from the server's own IP instead of the client's IP) for the Tridion website.

Because the server connects back to itself you would normally want to disable loopback checking (unless the server connects to itself over it's computername) and verify that the server can "ping" itself over the website's HOST header.