Factors Affecting ROI of Test Automation Some of the factors that influence the ROI of test automation are –
Test Automation Tool License cost
Number of test cases that could be automated
Time and Cost involved in initial designing of automation framework, feasibility and reskilling of team
Per hour cost of Automation Engineer
Number of builds/releases per year
Average number of test cases in each build
The configurations and regression cycle of each build.
Hardware, Software and other miscellaneous cost
Product Stability
Advantages of ROI Calculation
A firm knows if the effort and money used by the Automation team are paying off or not
Efficient test coverage.
Enhanced product quality
Feasibility of Quick execution in multiple environments
Focus on results and process improvements
Calculating ROI:
ROI is calculated as savings gained by replacing manual regression tests with automated tests divided by the cost of investment into test automation:
ROI = Savings ÷ (Investment + Maintenance)
Cost Savings Savings is the difference between the cost of running a set of tests manually versus running the same tests automatically a number of times over some period.
Savings = (time to run one manual test case - time to run one automated test case) * # of tests * # of runs
Investment Investment is the summation of fixed and ongoing costs of test automation including time spent building and configuring the test automation tooling or framework, and time spent coding or maintaining automated tests.
Investment = time to build framework + (time to code one automated test * # of tests) + maintenance costs
If there are purchase or licensing costs, they can be added here after converting dollars to minutes by dividing by bill rate as discussed above.
Maintenance costs are the costs of investigating and fixing failed tests (both false positives and actual failures). Flaky tests and development bugs alike contribute to the maintenance cost, as investigation is required to determine if the feature or the test is at fault. The more frequently tests fail, the longer it takes to break even on ROI.
Maintenance costs = maintenance time for one failing test case* % of failing test cases per run * # of test cases* # of runs
For your application it may not be possible to replace all manual tests with service or integration tests, but we consider it good practice in accordance with the Test Automation Pyramid to avoid relying fully on E2E UI tests, and to emphasize service and integration tests as much as possible.
Using this ROI model, business decision makers can quickly and easily understand the value that test automation delivers over time, while technical team members gain insight into the effects of test speed, robustness, and maintainability on the value of their test framework. Both can use it to drive right decision making, leading to high value test automation in your organization.
Another good example is given in the below link, also contains more details about maintenance costs.