# The unexpected costs of test automation maintenance and how to avoid them Updated on November 19, 2024 Lauren Harold ## A quick primer on test maintenance When you push a code change to your app, it can break your tests. For example, if you change your “Sign in” button labels to “Log in” button labels and some of your tests are still looking for “Sign In” buttons, then those tests will fail. These broken tests present false-positive failures, because the failures reflect a problem with the tests, not a problem with your application. These failures can result from any sort of change to your application that isn’t represented in your tests. **Test maintenance is the work done to update automated tests to reflect the latest, intended version of your app.** The more you update your app, the more someone needs to either (a) proactively update your tests or (b) investigate and then fix your failing tests. If you don’t perform regular maintenance, your automated test suite will quickly fall out of date. Then it won’t effectively do its job — finding bugs and other issues in your app. As the data will show, test maintenance can represent a time-consuming chunk of your automation efforts. And, between finding the root cause of test failures and then updating test automation scripts, it’s no-one’s idea of fun. It can take your engineers away from shipping code, and it often bottlenecks release pipelines. ## The data: How much time does test maintenance actually take? Creating and maintaining tests in open-source automation frameworks like Selenium, Cypress, and Playwright can be a drag on agile development teams aiming to ship fast and frequently. (Selenium is notoriously the most painful to maintain.) They all require someone with technical skills — like a developer or QA engineer — to dig around in code to find the offending part(s) of the test to fix. In fact, software developers often deliberately neglect to update their end-to-end (E2E) automated test scripts in favor of delivering more code. After all, most devs are incentivized to ship, and software teams have to ruthlessly prioritize their time and resources. No- and low-code test automation tools can make maintenance a bit easier, but they still carry maintenance costs. (More on that, shortly.) **According to our survey results, of the teams using open-source testing frameworks like Selenium, Cypress, and Playwright, 55% spend at least 20 hours per week creating and maintaining automated tests.** But that’s looking across all teams. In practice, the more complex your app is (measured by the number of non-QA engineers on your team), the more testing and test maintenance you’ll need to do. And, at least for the moment, the data show that using AI doesn’t provide any relief for teams automating tests with open-source frameworks. In fact, open-source teams using AI spend slightly _more_ time on test maintenance! ## What about using no-/low-code tools? **The data show that using an intuitive, no-code tool like the Rainforest QA platform is up to 3x faster than using an open-source framework.** Teams using no-code create and maintain automated test coverage in much less time than teams using open source. For teams using open-source frameworks, the survey data showed that small dev teams in particular struggle to keep their automated test suites up to date. (Probably because small teams are the least likely to have the necessary procedures and policies in place. When maintenance isn’t enforced, it can get ignored in favor of shipping code.) **Because it’s easier to create and update tests in a no-code tool, small teams are more able to keep their test suites updated, which makes the suites more reliable.** See more new research on the test automation technologies and practices that increase shipping velocity. **[Download the full report.](/content/state-of-test-automation-2024/index.html)** Of course, even when you’re using an optimal no-code tool, the cost of test maintenance is non-zero. Someone(s) on your team will still need to dedicate some number of hours every week to keeping tests updated. **If anything, the tendency to underestimate the costs of test maintenance is worse when a software team is adopting a no- or low-code solution.** Just because a tool is easier to use — much easier to use, in some cases — doesn’t mean the costs of maintaining automated tests go away. We saw this struggle all the time over the several years we sold a no-code test automation tool (before we enhanced it with AI and test automation services). We even have customers who signed on for our [test automation services](/content/test-automation-services/index.html) because [they couldn’t stay on top of maintenance](/content/case-studies/push-security/index.html) themselves with the version of our easy-to-use, no-code tool lacking AI. **It’s also important to note that not all no- and low-code tools are created equal** — some are much more complex than others, making them more difficult to learn and use. According to our customers, TestComplete and Katalon are among the most complex. (On the other hand, you can read for yourself [what they have to say about Rainforest](/content/testimonials/index.html).) ## How to avoid the costs of test maintenance Clearly there are significant time costs involved in test automation maintenance. And if you [don’t have a QA team](/content/blog/no-qa-team/index.html), there are other ways you want your engineers to spend their time and attention. So here are the ways you can automate your end-to-end tests while keeping maintenance to an absolute minimum. ### 1. Use a test automation tool that makes maintenance easier As we’ve established, test maintenance is generally easier and faster in no-/low-code tools than it is in open-source frameworks. But “no-code” isn’t the same as “intuitive,” so make sure you’re choosing a solution that’s as easy as possible for your team to learn and use. ### 2. Let AI handle maintenance for you There are several ways in which [AI-powered testing](/content/ai-accelerated-testing/index.html) in Rainforest QA helps you avoid unnecessary maintenance efforts and increase your release velocity. #### Reliably identifying elements in your app In most automated testing tools and frameworks, a slight change to an element in your app — like a button or form field — can break the relevant tests. #### Automatically updating tests When you make intended changes to your web app’s appearance or functionality, Rainforest’s AI can often automatically update — or “heal” — the relevant test steps so you don’t have to. #### Determining the app changes that matter Rainforest’s visual validation algorithm has been trained via machine learning to simulate the judgment of a human tester. ### 3. Write fewer, shorter test scripts This is a core component of the [testing best practices](/content/blog/qa-testing-best-practices/index.html) and methodologies we recommend to our customers. ### 4. Don’t use E2E test automation for new features still in active development Unit testing and integration testing are useful (and sensible!) early in the software development life cycle. ### 5. Look for opportunities to re-use test snippets Most test automation frameworks and tools will offer a way to create “snippets” of test scripts that you can re-use across different tests. ### 6. Make debugging easier with detailed test results Part of the pain of the test maintenance process starts with debugging — figuring out what needs to be fixed. ### 7. Use a done-for-you test automation service If you want your team to avoid the costs of test maintenance altogether, Rainforest QA’s [test automation service](/content/test-automation-services/index.html) will handle 100% of test creation and maintenance for you.