The Need for Speed and Quality
The slow speed of feedback to the dev team through the results of manual tests decreases productivity significantly. Re-executing manual tests in every iteration of SDLC is not a sustainable pattern in the current world. There is never enough time, and adding more personnel to execute manual regression tests leads to reduced returns. Test effectiveness is a critical aspect to keep up with the faster-paced development life cycles so that sufficient quality of the system can be ensured and time and money can be saved. To have better test effectiveness, we need to think differently with the increasing adoption of Agile and DevOps practices. We need to have automations in all layers of the test pyramid. This includes unit tests, component tests/service tests, and UI tests.
Finding the Right Set of Tests
The number of new features naturally causes the number of tests to increase significantly after a few iterations in almost all agile teams. The only way to keep up with the needed regression testing is to automate the right set of tests to ensure the change hasn’t impacted existing functionality. Ideally, we can’t test everything, and we can’t automate all our tests. So, we need to find the right balance. To accomplish this balance, mature DevOps teams use a combination of test automation and manual exploratory testing, both running in a continuous pattern.
Test automation implementations in different layers of the test pyramid are the following:
- Unit tests form the bottom layer of the pyramid with more test coverage
- One layer up, we have functional API tests or component tests
- UI tests form the top layer.
Additionally, we need to include performance and security tests in parallel to ensure better customer experience, speed, and reliability of our application.
- Performance testing via the UI or API
- Security testing (SAST and DAST)
In addition, there are tests that are extremely valuable when conducted manually. We need to continue these tests in parallel with test automation.
- Exploratory testing: Unscripted tests where the tester analyzes different aspects of the system with or without a prescribed end result. This helps to find new scenarios which can potentially identify new defects which may not be covered by automated tests.
- Usability testing: End-users/stakeholders/Ops teams are asked to test specific aspects of the system and to give feedback as they progress. This allows the team to better understand what users are thinking when they use the system.
What Is Continuous Testing?
Like continuous integration and continuous deployment are adopted in DevOps practice to remove bottlenecks in the delivery pipeline, project teams can improve their ability to test each valid software build as it becomes available. Continuous testing relies on test automation integrated as part of a deployment process where software is validated in realistic test environments. Continuous testing means adopting the right set of automated tests along with service virtualization, which allows the team to simulate missing services and environments so they can start testing earlier and more frequently. With continuous testing practices implemented, teams can become more efficient and effective. This can eventually enable the teams to reduce their costs as well as decrease the time it takes to get high-quality, innovative software to its end users.
Shifting Left Tests in the Pipeline
Having the right set of automations for the application under test is the main part of the continuous testing practice. CI/CD integration of these automated test suites (API tests, component tests, UI tests, and performance tests) helps us to achieve the shift-left model and then the continuous testing practice. This helps the teams to test early, test quite frequently, and test automatically in SDLC. Here are the key elements of continuous testing practice.
- Testcase management
- Test automation with maximum coverage
- Test environment management
- Test data management
- Service virtualisation
- CI/CD integration of test automations, also known as shift left
- Test insights and analytics through reports or dashboards
Shifting Left Security in the Pipeline
Most of the teams and organizations worldwide have already taken advantage of DevOps methodologies. However, software security is still lagging behind. Most of the time, the dev team doesn’t have the necessary tools to avoid problems in the first place. Usually, vulnerability checks are executed at the end of the development process in a waterfall fashion. Pen tests or vulnerability checks usually generate heavy documentation and sometimes force large sections of code to be rewritten. This approach creates friction between teams. While developers are trying to release quickly to deliver value, a waterfall approach to security ends up slowing this process down. Tests for reported vulnerabilities, in this case, are most likely manual and can take a long time to finish. This situation creates another nightmare if the software release date is already nearby.
Identifying the vulnerabilities during the initial stages of the software development process and the quick subsequent steps to tackle these can significantly help to reduce the overall cost associated with developing and maintaining the application. If these vulnerability checks are automated, it would help to flag the issues as early as possible and this can accelerate the delivery of these changes quite efficiently as possible. To identify security vulnerabilities at various stages, organizations can integrate various tools and services (cloud and third-party) into their DevSecOps pipelines. Please be mindful that integrating various tools and aggregating the vulnerability findings can be a challenge if we need to do it from scratch. But once implemented, organizations and agile teams can leverage the learnings and the process for the various already ongoing and upcoming projects inside the organizations quite easily.
General DevOps has introduced processes like continuous integration, continuous delivery, continuous testing, continuous availability, continuous monitoring, and continuous insights. These processes ensure the active building, deployment, testing, availability, and analytics of the application that we develop. Similarly, DevSecOps injects active security audits and penetration testing into agile development. DevSecOps advocates that security should be built into the product, rather than applied to a finished product. In reality, DevSecOps promotes traditional security engagement to the active process of the SDLC.
The most popular application security testing tools that DevSecOps teams implement in their development cycles are static application security testing (SAST) and dynamic application security testing (DAST). Understanding the differences between these and knowing when to use these are crucial to implement and enhance our DevSecOps strategy in order to secure our web applications.
Static application security testing (SAST) is a white box method of testing. This helps us to detect violations in our codebase, detect security vulnerabilities in the code that we own and on the libraries that we import. Modern SAST tools integrate well with the continuous delivery pipelines. We need to ensure that we choose a SAST scanner that is compatible with the programming language that our team uses. SonarQube and Checkmarx are some leading tools in this segment.
Dynamic application security testing (DAST) is a black-box testing method that examines our application while it is running to find vulnerabilities pretty much like what an attacker would do. DAST scanners may not have a dependency on specific languages since they interact with the application from the outside. OWASP Zap and Burp Suite are some tools in this segment.
The most important thing is to include both SAST and DAST in our security strategy, as these both bring their unique benefits to the table. We can have early feedback in order to secure our applications, once both these approaches are integrated well with the CD pipeline.