Automated CME functional validation after upgrade, hotfix or patch installation

Question

We are planning to automate our test cases used to validate that Tridion CME is functional as usual or not after any upgrade, hotfix or patch installation. We have good count of GUI extensions, customize events system, workflows etc in place. Below are few example test cases –

(1) To verify that workflows are actively running.

(2) Verify if deployer instances are up and running.

(3) Validate that the user is able to perform checkin / checkout / undo checkout / save/save-close /localize /unlocalize etc.

(4) Validate that admin is able to modify the user permissions.

(5) Validate whereused and tridion search is working fine.

(6) Validate our GUI extensions and other customization are functional as usual.

We did some analysis on that and come across below approaches and their challenges –

Approach 1- Selenium IDE for browser automation - it looks that it does not works in case of iframe and Tridion CME screens have good count of Iframe.

Approach 2- Core Service Solution to validate such test cases.

Approach 3- HP UFT automation tool.

Approach 4- PowerShell Script.

If anybody in this community have similar experience please share in detail.

Thanks in advance.

Answer 1

I would go for CoreService for all actions, apart from parts where you have GUI extensions. It's quite quick to write and easy to check. Running these tests will quickly show you the problem. Make sure your tests are able to run multiple times without too many preconditions (for example it generates unique names, independent on each other, few dependencies on environment state). However, the most problems during update are in CME area.

Next to it you might want to write some GUI automation tests. For this you may indeed use Selenium, only not Selenium IDE, but Selenium WebDriver. You will not be able to click and do other stuff with the default set, but there's very useful feature - ability to run JavaScript code. So instead of command to click on something, you just have a scriplet which says where to click and what to do. You will have to do quite some coding, but it will work. You should be able to catch the script bits with FireBug, or Chrome developer dashboard when doing the same actions in CME. So your test set will have urls and scriplets as input. You may then verify the actions made with CoreService, or Selenium. But also, keep in mind that this internal Tridion stuff and the JavaScript will most probably change from version to version.

The other approach to GUI automation is Visual Studio Coded UI, but it's quite expensive and will also require you to do a lot of coding (although you still might record some bits).

Answer 2

We've recently done an in place upgrade from 2011 to 2013 SP1 and used a combination of automated and manual tests.

1: For most of event system related changes Selenium works well. 2. For coreservice related applications you can write integration tests (writing them is expensive as it will take time as it is similar to writing your application and plus you got to maintain them). I will advocate for writing suite of integration tests that you can run whenever there is an upgrade as it helps a lot not just for upgrades but for regression testing when you continue to work on application development. You can test publish, checkin-checkout etc using coreservice integration tests and it helps automate most of the things you've outlined.

Last but not the least, there will always be aspect of manual test depending upon environment, setup and architecture and some issues which may crop up based on your setup and usage. Like we faced search related issues after upgrade. We had have master-slave configuration in search and we faced issues after our upgrade to 2013 SP1 which Tridion has now included in KB

http://tridion.kb.sdl.com/#tab:categoryTab:crumb:2:artId:5353