Random timeouts in transport phase when publishing

Question

In our new SDL Web 8.5 installation, we're seeing random publish failures.

these failures appear to happen in the transport phase.

In our transport log we can see the following errors

2017-08-24 11:33:46,798 ERROR BaseHTTPSTransportConnector - Unable to execute File retrieval on destination: http://*****:8084/httpupload
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
    at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:230)
    at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:424)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at com.tridion.transport.connection.connectors.BaseHTTPSTransportConnector.batchRetrieve(BaseHTTPSTransportConnector.java:299)
    at com.tridion.transport.connection.connectors.DiscoveryServiceTransportConnector.batchRetrieve(DiscoveryServiceTransportConnector.java:130)
    at com.tridion.transport.connection.connectors.TransportPoolConnector.batchRetrieve(TransportPoolConnector.java:85)
    at com.tridion.transport.DestinationController.poll(DestinationController.java:264)
    at com.tridion.transport.DestinationController.run(DestinationController.java:223)
    at java.lang.Thread.run(Unknown Source)

From the error, it looks like the transport service can't get a connection from the pool.

I've been trying to play with the number of workers and number of threads for publishing and deploying, but to no avail. Does anyone know what might be causing these timeouts ?

Answer 1

Putting some details here, just so some one can get help from it.

I recently found some publishing issues witth 8.5 and we fixed them with Hotfixes. in 8.5, we have some improvements in both Transport and in Deployer.

So the issues you are getting are not only with the Transport but you might also get them in Deployer.

The early symptoms were what you have pasted above.. and some more of them were:

• Transport failed: Could not transport tcm_0-5351810-66560.Content.zip using HTTPS
• Unable to establish transport to destination.Error while trying to reserve spots!
• ERROR TridionCommitStep - Commit failed for execution id:tcm:0-5352068-66560
• Transaction (Process ID 77) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

There are HotFixes out there for 8.5. Just contact SDL Support they should help you with finding them.

CD-3765 {contains cd_transport.jar} and 
CD-1047 {contains query validation and deployer updates})

Also, you might have to run below for Items not getting deleted from State store and for DB indexes.

Stop all services;

Remove items form EXECUTION table:

DELETE FROM [Broker].[dbo].[EXECUTION]

Check if all the constratinst are there for following tables:

EXECUTION TABLE:
CONSTRAINT "PK_EXECUTION_ID" PRIMARY KEY ("EXECUTION_ID")

PIPELINE TABLE:
CONSTRAINT "PK_PIPELINE_ID" PRIMARY KEY ("EXECUTION_ID", "PIPELINE_ID"

STEP TABLE:
CONSTRAINT "PK_STEP_ID" PRIMARY KEY ("EXECUTION_ID", "PIPELINE_ID", "STEP_ID")

EXECUTION_PROPERTY TABLE:
CONSTRAINT "PK_EXECUTION_PROPERTY_ID" PRIMARY KEY ("ID")
CONSTRAINT "FK_EXECUTION_PROPERTY" FOREIGN KEY ("EXECUTION_ID", "PIPELINE_ID", "STEP_ID")

If yes, run below:

ALTER DATABASE Broker
SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE Broker
SET READ_COMMITTED_SNAPSHOT ON

Restart services and try publishing again.