random stored procedure errors during bulk publishing

Question

We are currently upgrading a web 8.5 environment. Publishing is working fine, except when we do a bulk publish (we publish a large number of pages or components at the same time). Then, we randomly get database related errors. In the publish queue we see errors like

"A database error occurred while executing Stored Procedure "TCM_APPDATA_READBULK"

or

A database error occurred while executing Stored Procedure "EDA_ITEMS_READ".

These appear to occur randomly. Upon republishing the failed item, the item publishes correctly.

Unfortunately, that's all that the error says, i don't find any more information regarding the error. Strangely enough, we don't see this issue on a lower environment in our DTAP setup.

Does anyone know how to debug this error further ?

If it matters, we're using SQL Server on Amazon RDS.

UPDATE

Today, we don't get (until now) any errors regarding stored procedures, but we do see the following (possibly related) error appear in the event log. Note that this error doesn't reoccur when we republish the failed item.

Error downloading file 'tcm:11-20121' from 'DatabaseBinaryContentStorage'. See inner exception for more details.
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.)
The specified network name is no longer available


Component: Templating
Errorcode: 854
User: NT AUTHORITY\SYSTEM

StackTrace Information Details:
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Byte[] buff, Int32 offset, Int32 len, Int32& totalRead)
   at System.Data.SqlClient.TdsParserStateObject.TryReadPlpBytes(Byte[]& buff, Int32 offst, Int32 len, Int32& totalBytesRead)
   at System.Data.SqlClient.SqlDataReader.TryGetBytesInternalSequential(Int32 i, Byte[] buffer, Int32 index, Int32 length, Int32& bytesRead)
   at System.Data.SqlClient.SqlDataReader.TryGetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length, Int64& remaining)
   at System.Data.SqlClient.SqlDataReader.GetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
   at System.Data.SqlClient.SqlDataReader.GetBytes(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
   at Tridion.ContentManager.Data.AdoNet.ContentManagement.ItemDataMapper.GetBinaryContentPost(Stream stream, TcmUri id, IDataReader spOutput)
   at Tridion.ContentManager.Data.AdoNet.ContentManagement.ItemDataMapper.Tridion.ContentManager.Data.ContentManagement.IItemDataMapper.GetBinaryContent(Stream stream, TcmUri id, Int32 binaryId)
   at Tridion.ContentManager.DatabaseBinaryContentStorage.ReadBinaryContentToStream(TcmUri itemId, String blobId, Stream outStream)
   at Tridion.ContentManager.DatabaseBinaryContentStorage.ReadBinaryContentToStream(TcmUri itemId, String blobId, Stream outStream)
   at Tridion.ContentManager.ContentManagement.BinaryContent.WriteToStream(Stream binaryStream)
   at Tridion.ContentManager.ContentManagement.BinaryContent.GetByteArray()
   at CMSTemplates.PublishBinaryImagesInTemplate.PublishImages(Engine engine, String text, Package package)
   at CMSTemplates.PublishBinaryImagesInTemplate.Transform(Engine engine, Package package)
   at Tridion.ContentManager.Templating.Assembly.AssemblyMediator.Transform(Engine engine, Template template, Package package)
   at Tridion.ContentManager.Templating.Assembly.CSharpSourceCodeMediator.RunTemplate(Engine engine, Package package, String templateUri, String className)
   at Tridion.Templating.CSharpTemplate.CSharpSourceTemplate.Transform(Engine __engine, Package __package)
   at Tridion.ContentManager.Templating.Assembly.CSharpSourceCodeMediator.Transform(Engine engine, Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.ExecuteTemplate(Template template, Package package)

Answer 1

Because of the stored procedure errors, we were focusing on the CM database, but after a series of tests (we switched to a different database and got the same issues, and pointed a working CM to the 'broken' database and the issues were gone), we could prove that the issue was related to the CMS installation itself.

So, we started to look for any issues, and after some searching, we found that the installed .NET version was different. Our working server was using .NET 4.6.2, and the failing server was using .NET 4.5.2.

Since our custom .NET Tbb's were built against .NET 4.6.2, installing this version of .NET fixed the problem.

Interestingly enough, this difference in .NET versions caused the tbb's only randomly to fail, especially when publishing a large amount of items.

Looks like the issue reappeared after some time, so we'll need to dig further.

After a lot of debugging, we concluded that the problem must be related to connectivity or network issues between the CM server and the CM Database. Since we were running AWS instances of our server, we created a fresh instance of our CM server, did a reinstall, and the issue disappeared. We are still not entirely sure what caused the problem, but at least it's not related to our tridion configuration or our database.

Answer 2

I would suggest making contact with support to see if there's a hotfix. This one might be relevant, but you'll have to check.

CRQ-3922 Web 8.5 - Bulk publishing failing

The problem description for this is

Bulk publishing failing due to ORA-00060: deadlock detected while waiting for resource