About $2.41 billion worth of poor quality software costs in the U.S. alone. This was stated by the CISQ Consortium for Information & Software Quality in its report on “The Cost of Poor Software Quality in the US: A 2022 Report.” Among other facts, the document also addresses technical debt. Developers still lose about 33% of their productive time due to bad code. With a 40-hour week, that’s more than 13 hours.
But how can these numbers be reduced? What can be done to keep the quality of the software permanently high and to detect errors at an early stage? One tool in modern software development is test automation.
Test automation: definition, benefits and tools
The term “test automation” covers a large area. In general, this is the testing of software with the help of tools or other software. This is always done in three steps:
- Meet precondition
- Test execution
- Result validation
Test automation takes place in different stages of a test cycle. For each level there are different tools that can be used in a meaningful way.
A look at the test pyramid helps to clarify this:
In the lowest level, the component test or also called unit test, the individual processes and partial functions are tested. Tools such as Junit or comparable libraries for other languages are used here.
One level higher, the components are assembled and integrated with each other, and an exchange takes place between individual interfaces. A well-known test tool for this is the Citrus Framework for message-based systems.
Stage three then involves testing on the system. The complete system is assembled and a function test is performed.
Assuming that the system uses a database as persistence layer and the queries on it are made via an interface, an important criterion here is also a reasonable response speed. The requested data should be returned quickly, even if a large number of users access the process simultaneously. It is therefore also recommended to test load and performance. A well-known tool for this purpose is Gathling, for example.
At the top of the pyramid are the acceptance tests. In theory, the customer tests along here and checks the application. In practice, this is often solved by automated surface tests, as the clients do not have the time to click through all the masks. One of the best known tools for surface testing is undoubtedly Selenium. This makes it easy to test web applications in an automated way. Alternatives for desktop applications are Silk4J or Tosca.
Motivation for test automation
Manual tests are very tiring and monotonous. After the tenth test of the same procedure, everyone should quickly run out of steam. This can affect not only the subject’s motivation, but also the quality of the results. The attention decreases and one or the other defect could remain undiscovered. Not to mention that qualified personnel certainly have more important tasks than clicking the same mask over and over again.
But there is still the advantage of saving time. Test automation scripts execute tests much faster than any human. Thus, it is possible to perform more tests in a shorter time and thus achieve a higher test coverage. In addition, load tests, for example, are very difficult to perform manually. Simulating several thousand users is an expensive task. However, with automation scripts this is much easier. Testing non-functional requirements in particular, such as response times with a large number of simultaneous users, is often only possible with test automation.
Activities in the test procedure that can be automated
In principle, almost everything can be tested automatically, the question is more about the usefulness:
Does it make sense to automate a test case that is only executed once or twice?
If the complexity is minor, this may well be worthwhile. The effort is compared to the benefit and this also results in the spectrum of automation.
Important program paths, such as e.g. the login process in the application that need to be tested all the time are excellent candidates to start test automation. It is advisable to start here with tests that are not too trivial, but also do not affect the running operation and then expand the project step by step.
With automation, manual activities are replaced by scripts, documentation is created automatically and logs are generated. Ideally, this is done at the push of a button so that as little manual effort as possible is required. For more information, see Integration into the system landscape. But what might the actual architecture of such an automation solution, or TAS (=Test Automation Solution) for short, look like?
Universal architecture for test automation
The architecture of a test automation solution is versatile and can vary slightly depending on the use case. Nevertheless, they all have overlaps at their core. Interfaces always occur in some form, be it to external data stores or a simple GUI that is test automated via http. In addition to the interfaces, there is a form of execution layer in which both program code and the generation of logs and protocols take place.
Above this is a kind of definition layer that specifies test cases and preconditions, such as opening the browser. A simple example of this layer would be the given-when-then test case specification of Cucumber:
Required external libraries should also be integrated and managed here. The last layer in model-driven testing can be a generation layer, in which the required model is specified.
Not every layer is necessary for every type of test. For example, unit tests can do very well without interfaces.
Test case creation
Test case creation requires good communication with the business department. This is usually where the detailed knowledge about the system under test is located. Subject matter expertise provides some form of guidance on what steps are necessary to test, whether it’s a written specification, a click-through guide, or even a demo video.
These steps can then be mapped in the test case, where test data is used again and again. The integration possibilities are manifold:
- via a data table in the code (see given-when-then test case specification of Cucumber).
- integrated via external data storage via interface
- via a simple file that the system reads in beforehand
Test data and test script creation
In the case of test scripts, a decision must be made as to whether generation should be carried out using so-called record & replay tools or whether it is appropriate to code out the test cases.
Record & Replay records a user operation and then translates it into code. The method is very simple and can be used by anyone, but has the disadvantage that the code is very inflexible and often has to be regenerated when changes are made to the test system.
Considering that several hundred test cases with regular adjustments are used, this is rarely the optimal solution.
Coding out test scripts involves programming that is comparatively complex, but these scripts are much more sustainable and easier to maintain.
Test execution and test reporting
Once the first test cases are ready to be applied, the question of the execution cycle arises. While unit tests are executed in a few milliseconds, surface tests can take several minutes. If the number of tests is in the hundreds or even thousands, quite a bit of execution time is required. This is also the reason to introduce a suitable strategy. For example, do surface tests only run when GUI changes are also checked in? Do the overall tests run overnight? Numerous factors play a decisive role here and the right strategy is crucial.
The topic of documentation is also enormously important. Usually, reporting is generated after each test execution. Depending on the target group, this report should also contain suitable information. For developers, the stack trace is certainly important, for the project management rather how many percent of the tests were faulty and how this number has changed compared to the last report. A dashboard can also provide relevant information that every team member can view clearly and at any time.
Test administration and test maintenance
Once test scripts are created, they should also be maintained regularly. Test environments and the “system under test” (SUT for short) are constantly changing, especially during the development phase. Again and again parts are added, changed or removed. This procedure should also be used for test code. As soon as a test is no longer needed, it is advisable to remove it.
The same applies to regular maintenance. Program code must be kept up to date, otherwise errors can easily creep in and test results are no longer reliable. False negatives, i.e. reported errors caused by faulty test code, reduce confidence in test automation. A close exchange with the development team is an advantage here. In this way, the latter can best assess which program changes may have far-reaching effects on other parts.
Integration of the automated test procedures into the system landscape
With a large number of test cases, it becomes increasingly cumbersome to run all tests after code changes. Like the program code, the test code should also beintegrated into the existing Continuous Integration / Continuous Deployment(CI/CD) pipeline*. (* CI/CD is about the automated, continuous integration and distribution of new program code into the intended environment. Both are used together in the DevOps environment).
This means that the defined test strategy can be executed automatically without each team member having to think about it when making a new code adjustment (commit). Once stored in the build server, each code checkin starts the corresponding automated test.
All logs and reports are then generated and stored correctly in the desired locations for the various target groups. In order to provide information on the development status, further communications may be made, e.g. via e-mail to a specific group of people.
Advantages and disadvantages of test automation for software developers and companies
Some advantages have already been mentioned in the course of the article. However, in addition to the time savings and higher test turnover, there are a few more. This keeps employee motivation high, as they don’t always have to test the same processes. They can turn their attention to more important and meaningful tasks. Thus, established test automation can also save costs. Automation also allows test cases to be implemented that were not possible manually or only possible with a great deal of effort, such as load and performance tests.
However, there are of course drawbacks to test automation as well. Especially the initial costs should not be disregarded here. In most cases, an implementation requires programming experience and also regular maintenance. It may take several test cycles before automation is financially worthwhile.
The programming, technologies and tools used increase the complexity. Personnel with experience in these areas is necessary. Special care is required when developing the test code, as errors can creep in here as well.
Criticism of test automation and its approaches to solutions
The approaches to solving this problem are many and varied. From model-driven that then generate code to recording tools that generate test code afterwards as well. The goal is always to make it as easy as possible for the end user. In my eyes, however, this is only possible to a limited extent, as a certain degree of flexibility and experience is always required.
With a recording tool, the same parameters are always used for tests. But what happens when dynamic values, such as the time or date, become necessary? This trivial case is mostly still covered by building blocks. But what happens if you have to search for an entry in a table and said entry is always in a different place in the list?
As soon as this search result has to be clicked, many of the tools are either at the end or it becomes so complex that programming becomes necessary. Furthermore, there is the cost aspect. Commercial tools that offer model generation or recorders are often very expensive. The development environments that only offer programming are much more affordable here and a lot more flexible.
Writing a script that searches for a specific entry on a mask is, in my eyes, much easier than doing this via Record and Replay. Some test cases are simply too complex or are not used enough to justify the effort to automate them. In any case, the necessary know-how is required. Also, not every test is suitable for automation.
New role in test automation: the test automation engineer
For some time now, the role of the test automation engineer has existed in test teams. Whereas previously one tester took on all tasks equally, the current trend is more towards a division between test analyst and test automation specialist. While analysts focus on test preparation and test case creation, automation engineers are concerned with direct implementation as test scripts. The ISTQB has also long since recognized this, and since 2016 there has even been a separate certification for this: the Test Automation Engineer.
Conclusion: Test automation with enormous potential when used sensibly
Test automation is a very large area, this article is intended to provide a little insight about this topic. Numerous aspects have been addressed here, yet there are far more. The tools are extensive and the selection for the different areas is enormous. A separate article could be written about the different types of tests alone.
If test automation is implemented correctly, it can save an enormous amount of time, minimize errors and detect them at an early stage.
This is therefore unavoidable in larger software projects. Nevertheless, not all test cases should be automated across the board. Here, too, it is important to consider beforehand whether there is added value for the test case in question. Test cases that are only used once or are too complex to implement may not be suitable for automation. Reports should also be constantly evaluated to track changes and create metrics. How high is the defect rate, at which points do defects occur more frequently? Changes here can help uncover new areas of testing.
In summary, it can be clearly seen what advantages automation brings for companies and software projects, despite a high initial outlay. The use of the right strategy is crucial for the success of test automation. The right approach, tool selection and possible training of employees are important components. With these factors in mind, a test automation solution is a valuable tool for their business.