12

SDL Tridion 2011 SP1 HR1.

When I'm hitting the CME in the browser I see the GetUserSettings is stalling. It is taking 10 minutes - seriously... 10 minutes to receive a response. As detailed below, other TCM54 calls can be made following this and they are almost instant - resending the GetUserSettings request via Firebug takes 10 minutes again...

Watching the requests/SQL I note the following Initial request looks the same

The request on the live server is almost instant. On UAT (on the right hand side) the request is stalled waiting for a response… I can see (through SQL Trace) that the following query is sent to the CMS and returns the correct data almost immediately on hitting the URL: enter image description here

But the browser stalls calling the webservice and, as shown below responds after a lengthy period of time – nearly TEN minutes every time +/- 30 seconds… enter image description here

On following this through with Firebug I can see that the xmlHttp request is where the delay starts… I thought the JSON object being empty was significant this is the case in the working Live and problematic UAT instance. enter image description here

So eventually the CMS hits up… BUT… At this point on testing more, I can confirm that we’re unable open/amend Pages whilst I can still create and amend Components.

Pages are checked out and then stall in the following state (when opening an existing Page): enter image description here

I’m not seeing any errors/waiting calls in Firebug etc.

Oddly enough I can see other calls (GetLists etc) to the TCM54 Services are responding with a few ms as one would hope (these use the same xmlHttp request as above). This made me a little nervous about going straight in at the Network guys... enter image description here

Half way through this investigation I became aware of an error from the Transport Logs:

DEBUG RollingStateRetriever - Unable to read transport state file, not adding to summary batch

So I tested and can see that I am also unable to publish. I’ve confirmed the license and checked all services (rebooted the machine) and have also confirmed the cd_storage, deployer etc look OK (and as shown above, the SQL DB is being queried) – the error that eventually returns is:

The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:10:00'.

I’ve raised the issue with the client but they confirm no network changes have been implemented over the weekend - is there any way I can test-out/confirm this.

I’ve confirmed the COM+ services appear to be in order… enter image description here

So it's inconsistent in which calls, but consistent in the GetUserSettings call is always 10 minutes - and it's not clear why GetList and opening Components is OK whilst opening a Page fails .. or rather doesn't register a fail but simply doesn't complete.

Any pointers to an appropriate testing strategy to help nail what element(s) in the chain could be causing/contributing to this issue would be great.

Thanks

UPDATE: I tried the technique of disabling the Loopback Check (adding the registry DWORD DisableLoopbackCheck = 1 and rebooting) to no avail.

JAVA: (same version as live)

  • java version "1.6.0_20"
  • Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
  • Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

The same hotfixes are applied per the working LIVE version.

  • can you please confirm the version JRE on CMS server and CDA server and whether it is 32 bit, 64 bit or both – Pankaj Gaur Jun 3 '13 at 5:39
  • Also share the SDL Tridion Version - is it 5.3? – Pankaj Gaur Jun 3 '13 at 6:05
  • Good point Pankaj. SDL Tridion 2011 SP1 HR1 - I can't get on the machine to confirm JRE etc. but it is a replica of the LIVE system where this functions. – Dylan .. Mark Saunders Jun 3 '13 at 8:17
  • Interesting...From your analysis all looks good, but I will still emphasizes on the JRE version. The server may be replica of Live in configuration but JRE version automatic update configuration may result in different version of JRE. You from your end have analyzed almost everything, I will suggest, let's not leave the JRE and get the data for the confirmation. I will further suggest you to raise a Support Ticket and have a WebEx session with Tridion Support if possible. – Pankaj Gaur Jun 3 '13 at 9:27
  • Question updated Pankaj. I'm holding off SDL Support until I have confirmation that this is not a network issue as I'm aware some 'routing' updates where made late last week... will update the post when I have any info on that. – Dylan .. Mark Saunders Jun 3 '13 at 10:09
7

It seems the only answer we could assume was

an internal network update relating to internal DNS settings.

Unfortunately this is all the information we really have at the moment although we'll push to investigate what was changed (if anything, from what... to what...) as it's very interesting that the xmlHttp response would take 10 minutes every time and not for all TCM54 webservice requests...

If anyone else comes across this issue and can shed more light than the response above I'd be most appreciative ... but at least you should have a little more confidence going in at the network guys :)

We did try local testing such as tracert etc. but we could see normal HTTP requests resolving fine upto the point of the webservice call to GetUserSettings. As I say above, the GetList (to list SGs/Pages) call to the same webservice resolved in a few milleseconds - every time.

UPDATE: PROBABLE CAUSE: Something we'd discarded as we didn't see a pattern that would implicate this was that we have an external application that imports content into the CMS based on exports from and external system. This application executes every hour but the error we report only occurred once or twice a day and not at the same time.

On further investigation we did find a pattern over the days that the error was always occurring around the hour.

Investigation of client network activities etc. turned up nothing then the realisation that this could be related to our app!

app background Within the app we upload content using the Core Service and for binary items we create a new socket an push the binary item into the CMS. Something we should have realised from the outset was that the client isn't a robot and the content is updated as and when required - not consistently in the sense of every hour would be the same!

Alas... when the client DID release a batch with a number of binary updated, the application was simply iterating through binary items and pushing them in using up all available sockets -

enter image description here

thus, it seems, the CMS was unable to connect over the network... I'm not sure why this would only affect certain webservice calls unless these are using different communication methods in the back end (something it will be nice to investigate when time permits) - but now we've updated the app to utilise a single socket we no longer see the same issue.

  • Did you try a 'tracert' to your cms server from your client? – Chris Morgan Jun 3 '13 at 17:57
5

Not all requests are made from the browser to the webserver; for some requests the webserver connects back to itself. I would presume that in your case these are the "problematic" calls as apparently the browser could resolve the DNS name of the server without any problems. When the server->server connections take place in a blocking manner (probably going through some lengthy timeout) this will also block the AJAX request from the client.

You can see this in the IIS logs (the connection would be made from the server's own IP instead of the client's IP) for the Tridion website.

Because the server connects back to itself you would normally want to disable loopback checking (unless the server connects to itself over it's computername) and verify that the server can "ping" itself over the website's HOST header.

  • These additional testing steps will certainly be undertaken if this issue appears again. We can't replicate this now as we don't own the network or have access to the DNS settings etc. Thanks Bj0rn. – Dylan .. Mark Saunders Jun 4 '13 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.