Robot Framework CI/CD with Azure DevOps
CI/CD is becoming ever more important step in a release strategy within teams using Test Automation in their Agile processes. According to Forrester report, Software testing is a key part of the Agile+DevOps life cycle and 72% of firms say testers are critical to continuous delivery success. Importance of DevOps is visible with the rise of big players in this domain: Amazon Web Services, Azure DevOps Services, or SouceLabs as cloud testing platform. This trend of moving complete CI/CD pipeline to the cloud, means affordable and reliable infrastructure, better metrics, and easy-to-use GUI with lots of documentation. Having that in mind, understanding DevOps paradigm is a necessity for test engineers looking to jump on this train and more importantly, to have complete picture of their product release cycle, technologies and procedures in use.
This post will be centered around technical implementation of Test Automation in DevOps cloud platform, specifically, how to integrate your GUI automated scripts developed in Robot Framework with Azure DevOps pipeline. Our goal is to have Build process set up, which means to run tests on cloud agent and present test results into graphical form:
Before we dive in, you should have next prerequisites already set up:
- GitHub project with your Robot Framework Tests (if you don’t, you can use this “Hello World” project)
- Azure DevOps account, which you get simply by having Microsoft Account
- Basic understanding of CI/CD paradigm
- Some Experience with GUI testing and Selenium (it does not have to be Robot Framework exclusively)
So once that’s sorted out, log in into your Azure DevOps portal https://dev.azure.com/ and create new project, name it and choose Git as source control in Advanced Tab. Project basics are now settled down, we move to creating CI pipeline, and we will focus only on Build process.
And what do we mean by Build process exactly?
Imagine you have brand new installation of Win 10 or other operating system on your machine (Agent pool), and someone gives you USB stick with code with GUI Test Automation scripts (C#, Java, Protractor, Robot Framework). First thing you (Build agent) want to do is install Python on your machine or other interpreter needed for running your code. Next, through pip package manager you’ll have to install Robot Framework, Selenium library, and chrome driver.
And that’s what our build steps will be about. Installing Python and necessary libraries, chromedriver, and running scripts through Powershell commands.
Let’s start.
Click on Pipeline -> New Pipeline and select GitHub, enter credentials from your account and select repository with your framework/test scripts. Next, you’ll be asked to configure your pipeline, and Azure DevOps will present you with build steps that are most commonly used. We will Use Python package.
Now give it a moment and observe a default YAML file, which is basically a set of rules for an agent on cloud machine, with steps that will install Ubuntu OS, different versions of Python, pytest library, and steps for Publish job.
I’ll try explaining the YAML file we use, with only few steps needed for running the build.
Red Square:
-vmImage is what OS cloud machine is using. In this case we will choose Windows 10 with Visual Studio 2017 preinstalled. (vs2017-win2016)
- In default YAML, there is a matrix with different Python version to be installed, so select only 3.7 version, but you can leave to whichever Python version you need.
- UsePythonVersion@0 presents a step that will point which Python version will system use when running RF tests.
Green Square:
- Robot Framework and Selenium Library are needed to run tests, and we will install chromedriver with npm package, those are dependencies for project.
Yellow Square:
- Now our machine and OS are set up, with all libraries needed. Next is to run robot scripts with Powershell.
robot --pythonpath . -x outputxunit.xml TestCases.robot.(This project is using external python class, therefore — pythonpath flag in command)
All done!
Click Run button on right top of the screen and watch how the cloud agent is setting up your build steps and running Robot tests.
If everything goes smooth, all build steps are green, and in step with commands running tests, you can find your tests results. (Run Robot Scripts in this case).
Let’s check what have we done up until now:
- Created a project with code repository of RF tests
- Set up Build with desired agent (YAML file)
- Implement steps with libraries and and instructions to running tests
Next, let’s instruct the agent for following build step: Generating reports from RF, and displaying in more graphical and readable manner, as in picture from the start of the blog.
This means that we need another copy log.xml file which would be renamed to outputxunit.xml. Hence that in previous picture, we have both of those files generated, and they are basically same, with some extra information in outputxunit.xml. This file is created through Powershell command: -x outputxunit.xml.
As for the next step, you need two parameters: point to how your .xml file is named and location of the file. In most cases, it would be as in this case:
- task: PublishTestResults@2
inputs:
testResultsFiles: outputxunit.xml
searchFolder: 'D:\a\1\s\'
condition: succeededOrFailed()
displayName: 'Publish Test Results outputxunit.xml'Once the the step added to the end of YAML file, save and run the build again. Under the Tests tab, graphics with tests results will be displayed.
Under Test Plans>Runs, you can check the history of your previous runs:
Also, note that in RF script, the path for chrome driver should be the same as installation path in your build step. So in case of this project, following parameter will be added to Create Webdriver keyword:
#Open Webdriver hosted on Azure DevopsCreate Webdriver Chrome executable_path=D:/a/1/s/node_modules/chromedriver/lib/chromedriver/chromedriver.exe
In next post, we’ll go through how to integrate Jenkins and Robot Framework tests. Even though this post is just a scratch of what real DevOps means in practice, it could be helpful to get some understanding of how Continuous Integration could work with your Test Automation project.
Feel free to reach me for any advise:
milan.novovic@finallinetechnologies.com
