Test automation is only one part of the DevOps lifecycle, but critical for fast and high quality delivery. With automated testing, it’s possible to achieve continuous testing - an essential part of delivering quality software at high speed, and an important driver for increasing productivity, lowering costs and lowering risk.
Why automate testing in the DevOps lifecycle?
How to choose your test cases and build your test automation flows
How to find the right test automation tool for DevOps
Why automate testing in the DevOps lifecycle?
If you’ve read our blog post Why Test Automation is Critical in DevOps, you’ll probably have a good overview already of why test automation plays a key role in DevOps. If you haven’t, here’s a recap: DevOps goes hand in hand with agile practices and CI/CD. These concepts all have in common that they aim for increased flexibility and speed to allow a better flow through the release pipeline with fewer bottlenecks and faster feedback loops. Overall, the aim of these approaches is to deliver a high quality in product releases and updates, but at a much higher speed than more traditional models, such as the waterfall approach.
If performed manually, many steps in the release pipeline are slowed down, and errors are more likely to occur. Automation introduces speed and accuracy, making quality delivery at speed possible.
Testing is just one part of the release pipeline, but nonetheless an obvious candidate for automation. Particularly regression testing, which is highly repetitive and time-consuming, can with benefit be automated to free up tester resources, so that they can focus on work that requires their human expertise, instead of doing robotic work.
Faster test outcomes: Test automation can be implemented by writing a test script on a tool. The code in development is put through that script and checked for expected outcomes. Any deviation is noted and investigated for possible errors. This is way faster than manual testing, where QA specialists find repetitive testing frustrating and have a tendency to miss errors. Consistent results: Consistency in QA means the software behaves on expected lines throughout the delivery pipeline. Test automation can validate the code for variables at different stages and mitigate the chances of human errors. Ease with regression testing: When a major update is implemented, certain types of tests, such as usability testing, are performed manually. However, when there are more variables and the software needs to be checked across devices, operating systems, browsers, geographies and networks, then automatic testing takes the cake. It can work non-stop and check for deviations or non-compliance. Human resources utilization: During manual testing, scarce human testers are deployed in tasks that require constant attention. Further, to avoid boredom arising out of conducting repeated testing, QA testers can give certain tests a miss. Imagine the type or number of glitches escaping the test dragnet and later ruining the user experience. However, with test automation, repetitive testing can go unhindered wherein scarce human resources can be deployed for better activities.
How to choose your test cases and build your test automation flows Now that we’ve covered the nuts and bolts of test automation and DevOps, we’re ready to take a look at implementation. In order to implement automated testing into your DevOps lifecycle, you must first map out your release pipeline. This can be done by following four steps:
Identify all the stages of your release
Identify the gates and requirements necessary in a build’s journey to production
Identify the feedback mechanism required for quick error detection and solving
List all the operational procedures, services, and actions involved with a release cycle
The next step is building your automation flows. The ease and speed at which you can do this, as well as the approach you will need to take, depends completely on the tool you choose.
In addition to the right test automation tool, it’s ideal to have a test automation strategy in place, as well as to familiarize yourself with some best practices. Best practices for getting started with test automation are:
Build your automation flows gradually and increase coverage over time: Start out with flows that are easy to automate, and that are both predictable and repetitive in nature. In most cases, you will find that it’s the relatively simple and very repetitive flows that, by far, take up most of your testing time.
Start by testing one thing at a time: By limiting the complexity of a single test flow, it’s easier to detect what goes wrong when a test case fails.This is not only a best practice for beginners, but in general. Instead of bundling up multiple tests in one test case, it is best practice to build reusable components with your test automation tool. This way, it is easy to reuse the logic contained in other test cases, and the time required to create a new test case is minimized.
Build automated test cases that are independent and self-contained: You can reuse the structure of flows across test cases, but it’s a good idea to keep tests separate. This way, they can all be scheduled at once to be executed anytime and in parallel, i.e. across different environments.
Ensure collective ownership of test automation: Remember that success with automation is dependent on a team’s collective knowledge. Adopt a test automation platform that all testers can work with, so that automation becomes a natural part of the daily work of all team members.
How to find the right test automation tool for DevOps The last step in automating testing in the DevOps lifecycle is to find the right tool.
There are a vast number of options when it comes to test automation tools. The one you choose will impact how agile your DevOps lifecycle is. Only with the right tool for your team can you achieve quality delivery at speed, which is the goal for DevOps.
Popular tools include open-source options such as Selenium. It’s a good idea, however, to understand how these tools work before you invest your time and effort.
A DevOps test automation tool must:
Be usable by testers, developers, operations personnel, and management
Integrate seamlessly into your CI/CD pipeline/with CI/CD tools
Be able to run in any kind of infrastructure
Keep maintenance of automation cases at an absolute minimum
Not rely on users to write code, no matter how complex their automation needs are
Not take time away from testers’ primary responsibility; challenging the end-product
Have a very short learning curve to not disrupt release cycles
Make it very easy to collaborate on automation cases
The type of testing tools: This could either be open source or commercial, the intended platform, the language such as Java or C#, and whether the tools are for testers or developers.
Licensing and support cost: Costs determine what tools one can afford to buy depending on the budget. There are many available open source tools, such as Selenium; however, most open source tools require technical skills (i.e., programming skills and experience).
Whether the tool supports continuous integration (CI) and DevOps tool integration: A tool that doesn’t support CI or DevOps may not be effective, especially in agile projects where speed is the main concern.
The availability of training, tutorials, and documentation that will guide the users.
Good test reports: An insightful and informative test report will help analyze defects and root causes as well as test effectiveness among analyses.
Customer support, systems you are already using, and how you will maintain tests.
A deep understanding of your project requirements: This includes the project scope, project type (whether desktop, mobile, or web-based), and the skills of your existing team.
look at a common workflow to see how everything fits together and provides real value to the business:
A developer, test engineer and stakeholder meet to discuss a user story and come up with concrete examples of behaviors from a business standpoint. These behaviors are converted into executable behavior-driven development (BDD) tests.
Developers pull a user story from the backlog and write unit and integration tests (in conjunction with test engineers) as part of the test-driven development (TDD) process on their local machines. The tests and code are deployed to a shared code repository when everything passes.
DevOps engineers develop continuous integration (CI) servers that run code in a shared repository, execute all of the tests (TDD and BDD) and ensure everything passes. They may also setup different branches for pre-production or staging.
QA or automation engineers write additional tests, as required, and run other quality assurance processes. For example, they run performance tests to ensure that an application doesn't contain any bottlenecks under an expected load.
Ops engineers reuse the tests in the shared repository for acceptance testing and ongoing post-production monitoring.
Ten Test Cases You Should Automate 1. Repetitive Test Runs This is the golden rule. If the test run is repetitive, you should automate it. This will save your precious time and energy. In addition, if this is the test you assume that you will run frequently in the future, automating is a very good idea. You will have more accurate and precise results, each time.
2. High-Risk Test Cases If the involved risk is high, you should automate that test case. Prioritize the test cases based on human error hazard potential.
3. Critical Parts of A Web Application Automating critical parts of your web app is a smart idea. You want to avoid any possibility of man-made error and to be on the safe side. This is especially true for test cases that can jeopardize the whole web application.
4. Extensive Tests Inputting lots of data can be exhausting. In addition, the chances of making a mistake during the manual testing increase. This is why test which implies large data sets are usually automated. Mainly, we refer to those test cases which require filling long forms.
5. Evaluate the Pros and Cons By evaluation, we imply the time required to complete the automation process or to do the test manually. If you are not ready to invest time in the automation process, i.e. if the process itself would take more time than manual testing, common sense says that the automation is not worth it. This is especially true for non-repetitive tests.
6. Three Golden Tests There are three test cases that you should not question when it comes to automation: the regression test, smoke test, and sanity test. Why? Because each build or release of the app requires testing.
7. If You Can Answer the Magical Question “Why?” Don’t ever automate a test case just because you can. This could bring you more harm than good. If it will benefit the app, your team, or the organization, go ahead. Your goal should always be fast feedback, precision, and better overall results. However, don’t forget that Selenium requires deep knowledge when it comes to script writing. Manual testing is sometimes a better option, especially if you are not a skilled Selenium user. Remember that the answer to the question “Why?” when it comes to automation must never be “Because I can.”
8. Complex Cases Automating complex test cases is almost a rule. Again, you do want to be on the safe side and to save your time when it comes to complexity.
9. Performance Testing We usually choose to automate performance testing, such as load testing and stress testing. 10. Long Testing If a case requires an overnight devotion, you should automate it.