We have an issue where we need to clean up old Publish Transactions. Failed transactions should be deleted after some time, same is for successful one. When I take a look online, there are no out of the box solutions for this. Has anyone created GUI extension for this, or some custom tool that tracks old transactions and deletes them after some time?
Take a look at the following, Purging finished publish transactions
This allows you to delete success or failed transactions, you can even define how old the transactions need to be in order to be processed.
Of course you can always develop your custom solution if this is missing any advanced features you might need.
You could take a look at the Powershell Purge solution in 2013 using core service, I don't think much has changed in 8.5 Tridion 2013SP1 Purge Publishing Transactions with Powershell Problem