S3Corp. Tech Blog

Don’t Let Testing Stop Your Agility

Agile has transitioned from an ideal to a standard practice in businesses across the globe. What was once a cutting-edge methodology reserved for a select few is now a common way of working. Its goal is simple: to build higher-quality software faster. Agile enables businesses to adapt to rapid changes and provide value quickly, but achieving this requires effective testing practices that match the speed and flexibility agile demands. Without proper testing, the success of agile initiatives is at risk, as testing needs to be integrated into every stage of the development cycle.

The Role of Testing in Agile

Testing is integral to the agile process, not just an afterthought or a final task. As Eric Taylor, director of agile software delivery at AgileTrailblazers, points out, testing must be prioritized from the beginning. In an agile environment, speed is paramount, and testing must evolve to keep pace with rapid development cycles. “With the speed that agile brings, the only way to really achieve that speed is to actually get your testing in order,” Taylor says. “You have to really put testing first, and change testing to be the mindset of the entire organization as opposed to being this thing you do at the end.”

Organizations that neglect testing at the start of their agile transformation face significant risks. Without proper testing processes in place, bottlenecks emerge, slowing down development. Zubin Irani, CEO of cPrime, emphasizes that agile transformations are difficult and often painful when testing isn’t addressed early on. “Usually what happens is you go through a transformation, you realize it isn’t working, and then finally you address testing,” Irani notes. However, this delay can result in poor-quality software and unmet business objectives.

Wayne Ariola, chief strategy officer at Parasoft, highlights that testing is often the last consideration during agile transformations. Despite this, it is one of the most critical components of the process. Testing impacts everything from speed to quality, and ignoring it leads to significant problems. “If you look at agile processes, the biggest impact associated with the change is testing,” Ariola asserts.

The urgency of integrating testing into the agile process is further compounded by the growing demand for businesses to succeed in the digital space. In today’s market, where software is central to business success, agile and testing practices are essential for maintaining quality and speed. Thomas Murphy, research director at Gartner, acknowledges this shift, stating that businesses must adopt agile and DevOps practices to thrive in the digital world.

Adapting Testing to Agile Practices

In an agile world, testing is no longer a separate phase but a continuous set of activities. Ian McLeod, chief product officer at Rogue Wave, explains that the role of testing has changed dramatically. Testers no longer have the luxury of waiting until the end of the development cycle to test code. Instead, they must be part of the development process from the start, ensuring that quality is maintained throughout. “Testing is less a phase and more a set of activities now,” McLeod states.

To succeed in agile, the entire team must embrace a testing mindset. Taylor of AgileTrailblazers emphasizes the importance of creating a culture where everyone, from developers to product owners, prioritizes quality. “The role of the tester really being on the team is sort of the guardian of quality,” Taylor says. “Their job is to make sure the quality of the work everyone is doing is staying high.”

A crucial aspect of ensuring agile success is maintaining open communication between business teams, developers, and quality assurance. Brad Stoner, senior sales engineer at AppDynamics, stresses that for agile to work, the process must change from front to back, from business to development to QA. There needs to be a strong feedback loop, where QA is empowered to point out bottlenecks or potential issues with the software early in the process. “Unless we make this app testable, we are always going to be your bottleneck,” Stoner says.

In addition to communication and collaboration, teams must stop focusing on whether they are “done testing.” As Ariola from Parasoft explains, this question is more about sprint scheduling than the actual quality of the release. A better question to ask is whether the risk of the release candidate is acceptable. By shifting focus from completion to risk mitigation, teams can prioritize quality and business needs over arbitrary deadlines.

The Power of Automation in Agile Testing

Automation plays a significant role in improving the speed and effectiveness of testing in agile environments. Stoner suggests automating simple tasks to quickly verify whether the release is performing as expected. This allows QA to focus on more critical tasks and provides faster feedback to development teams. “Instead of having QA look at these bad releases multiple times a day, they don’t have to waste their time,” Stoner notes. This simple automation helps speed up the testing process without incurring high costs.

Automation is also essential for scaling testing efforts. According to Anand Kamat, group program manager at Microsoft, testers can automate a variety of tasks, including test environments, end-to-end scenarios, and app performance monitoring. Kamat suggests leveraging cloud environments and scripting skills to automate testing in agile teams. Automation allows testers to address repetitive tasks and ensure better coverage across different scenarios.

However, McLeod of Rogue Wave acknowledges that automating testing can be more challenging for organizations that are converting their processes rather than starting from scratch. For companies making the transition to agile, investing in several sprints is necessary to effectively implement automation. Despite the challenges, the benefits of automation are clear, providing faster and more consistent results.

Manual Testing: A Necessary Complement

While automation is crucial, manual testing remains an essential part of the testing process. Kamat points out that automation does not guarantee 100% coverage of use cases, and testers should avoid the false confidence that can come from automated test results. Exploratory testing is a key complement to automated tests. Subject matter experts can test real-world scenarios, focusing on user experience and usability, which automated tests may overlook.

Taylor agrees with the importance of exploratory testing, stating that it allows testers to explore scenarios that might not have been considered otherwise. “Exploratory testing gives an analytical individual the time to go through the site and think of all the kinds of odd scenarios that others might not have been thought of,” Taylor says. This approach ensures that edge cases and user behaviors are considered in the testing process, adding an additional layer of quality assurance.

Unit Testing: Ensuring Core Functionality

Unit testing is another critical component of the agile testing process. According to Ariola from Parasoft, unit tests provide immediate feedback on user stories, allowing teams to understand the impact of changes quickly. Without proper unit testing, quality tasks get deferred, leading to higher costs and potential failures later in the development cycle.

McLeod from Rogue Wave emphasizes the importance of unit tests in agile. “You don’t write code unless you have a unit test to test it,” he says. By writing the test before the code, developers ensure that each component works as expected, reducing the chances of defects and ensuring faster delivery.

Unit tests should be comprehensive, covering a wide range of inputs and expected outputs. McLeod compares writing a unit test to writing a specification for the component. Rather than relying on documentation, developers write tests to ensure that the component functions correctly. This approach reinforces the importance of quality throughout the development process, ensuring that each feature is built with testing in mind.

Static Code Analysis: Identifying Issues Early

Static code analysis is a proactive approach to quality that allows teams to identify vulnerabilities, defects, and security weaknesses before they become major problems. McLeod stresses the importance of addressing quality early in the development cycle. “Quality is cheaper if you do it early in the process because you can make sure your stuff is right the first time you check it in,” he says. Static code analysis tools help teams detect issues in their code before they break the build or cause other disruptions.

By integrating static code analysis into the development process, teams can reduce the cost and complexity of fixing issues later. Early detection of problems leads to faster resolution and less risk of failure during later stages of the development cycle.

Test-Driven Development: Prioritizing Quality

Test-Driven Development (TDD) is a fundamental practice in agile development that helps teams maintain high-quality standards. Irani from cPrime believes that TDD encourages developers to think from a testing perspective, leading to better quality code. By writing tests before code, developers ensure that they are building the right features and that they are testable from the start.

TDD is a powerful tool for improving software quality, but it can be challenging for testers to master. Irani notes that TDD requires maturity, but when done correctly, it results in higher-quality software and better collaboration between developers and software testers. “It puts quality first,” he explains. “It’s just prioritizing quality and allowing you to think about it in a different way because you had to think about it from a testing perspective.”

Independent Test Groups: Maintaining Objectivity

Having independent test teams is an effective way to maintain objectivity and ensure comprehensive testing. Black, president of RBCS, recommends that independent test teams should handle critical tasks like building and maintaining a regression test suite. These teams should focus on aspects like usability, security, performance, and reliability, which are often difficult to integrate into sprint teams.

Taylor from AgileTrailblazers agrees, noting that testers should be fully integrated into the development team, but there must be an independent group focused on testing system-wide properties. These properties are emergent behaviors that are difficult to capture in isolated user stories or features. Having a separate group ensures that these critical aspects of quality are not overlooked during the agile process.

The Testing Pyramid: A Key Principle for Effective Software Testing

The testing pyramid is a well-established concept in software development, introduced by Mike Cohn. This model divides testing into three layers: unit tests at the bottom, integration tests in the middle, and user interface (UI) tests at the top. This structure reflects the different costs, complexity, and frequency with which each type of test should be written. According to Taylor, unit tests are the easiest to write and should be the most numerous. These tests are designed to verify individual components or functions of the software in isolation, ensuring that they work as expected. Unit tests are quick to execute and can be automated easily, making them an ideal choice for the majority of a development team’s testing efforts.

In contrast, UI tests are the most difficult to automate and should be kept to a minimum. They often break due to changes in the user interface and can be slow to execute, making them costly to maintain. Taylor suggests focusing the bulk of your testing efforts on unit and integration tests, which are more stable and manageable. UI tests are still important but should be approached cautiously, with a smaller portion of resources allocated to them. Over time, UI tests can be built up, but they should always be secondary to the foundational unit and integration tests.

The role of integration tests, situated in the middle of the pyramid, is to verify that different components of the software work together as expected. These tests are essential for ensuring that the interactions between various parts of the system, such as APIs or databases, function correctly. Unlike unit tests, integration tests often require more setup and resources but are less prone to the frequent changes that affect UI tests. Therefore, the optimal strategy is to concentrate efforts on unit tests first, then integration tests, and finally, the UI tests.

Microsoft’s Kamat emphasizes the importance of service or API testing as a more reliable alternative to UI automation. By focusing on these areas, teams can better validate functionality without the inherent brittleness of UI tests. This approach also encourages exploratory testing, which can uncover unexpected issues and ensure the product works well in real-world scenarios.

Utilizing Beta Groups for Real-World Testing

One way to ensure that a product works as expected before it reaches a broader audience is by using beta groups for internal testing. Beta testing involves releasing the product to a limited set of users, who can provide valuable feedback on its performance and usability. John Basso, CIO and co-founder of Amadeus, explains that beta testing allows developers to respond quickly to issues, fix bugs, and address user confusion before the product reaches a wider audience. This early testing provides insight into how the product performs in real-world conditions, where unforeseen issues often arise.

Rather than rolling out a product to 100% of users at once, teams can opt to release it to a smaller percentage, such as 10%. This minimizes the risk of widespread issues and allows teams to fix any bugs or defects quickly before expanding the release. This incremental approach ensures that if something goes wrong, only a small subset of users will be affected, which facilitates faster recovery and problem-solving.

Testing Less but Fixing Faster: A Shift in Approach

In modern software development, particularly in agile environments, testers often face the challenge of testing less while still ensuring that the product is functional and stable. Rogue Wave’s McLeod points out that this approach may be suitable for fast, easy-to-update software that is not mission-critical. For example, applications that support e-commerce or content management systems may benefit from testing less upfront, instead opting to fix issues quickly once the product is live. This strategy can help deliver value to customers faster, as long as the software is capable of being quickly updated in response to issues.

However, McLeod acknowledges that this approach is not appropriate for all types of software, particularly mission-critical systems. If the software in question is responsible for essential tasks, such as keeping airplanes in the air or controlling medical devices, more rigorous testing is necessary. For these types of systems, a shift-left approach, which emphasizes testing early in the development cycle, is crucial to ensure reliability and prevent catastrophic failures.

Microsoft’s Kamat advocates for a shift-right approach, where testing continues in production, and real-time monitoring detects issues as they occur. While this doesn’t negate the need for early testing, it complements it by providing additional safeguards once the product is in use. The key is to detect problems quickly in production and fix them before they impact a large portion of the user base. By continuously monitoring the application after release, teams can address problems proactively and improve the overall user experience.

A Lesson from Amazon Prime Day

Real-world examples highlight the critical role of software quality. During Amazon Prime Day, one of the largest online shopping events of the year, the company faced a significant software glitch that prevented customers from adding items to their shopping carts. Despite extensive planning and preparation, even major tech companies like Amazon are not immune to software failures. As users took to social media to express their frustration, Amazon’s reputation suffered, demonstrating how quickly issues can escalate in the digital age.

Ariola from Parasoft stresses the lesson from Amazon’s Prime Day failure: software quality is no longer optional. Small bugs or glitches can blow up on social media, leading to negative publicity and potential financial losses. The cost of failure has risen dramatically, and companies must prioritize quality to avoid such public disasters. As social media amplifies the voices of frustrated customers, businesses must understand the long-term impact that even minor issues can have on their brand.

The Dangers of the “Fail Fast” Mentality

In agile development, there is often an emphasis on the concept of “fail fast, fail often.” While this mindset encourages experimentation and learning from failures, it can also be dangerous if misapplied. RBCS’s Black warns against adopting a fail-fast mentality without considering the consequences. While some projects, such as mobile games, may tolerate frequent failures, others, like self-driving cars or medical software, cannot afford to fail.

Black argues that adopting a mindset of simply “throwing things out there” with the expectation of fixing issues later can lead to disastrous outcomes. As software becomes more integrated into critical infrastructure, the stakes are higher. Rushing software to market without adequate testing and quality assurance can result in catastrophic failures. The true goal of agile development should not be speed at the expense of quality but rather a balanced approach that ensures both rapid delivery and high standards of quality.

The Evolving Role of Testers in Agile Development

In agile development, the role of testers has evolved significantly. In traditional software development, testers were often isolated from developers, focusing primarily on catching bugs in the final stages of development. In agile environments, however, everyone is responsible for quality, from developers to product owners to end users. As a result, testers must now broaden their skill sets to adapt to this new way of working.

Ariola from Parasoft suggests that testers need to understand the broader business context of the software they are testing. This means thinking beyond individual tests and considering how the software impacts the entire business. Testers must be able to see the connections between different parts of the system, including third-party services, APIs, and backend components. This requires a deep understanding of the application’s architecture and how it fits into the larger business ecosystem.

In addition, testers must become more technical. As automation becomes a central part of the testing process, testers need to be proficient in writing code and using automated testing tools. Automation is a critical component of modern software testing, and testers must have the skills to develop and maintain these tools effectively. Testers must also become comfortable working with Continuous Integration (CI) and Continuous Delivery (CD) platforms, which enable seamless deployment and testing of software updates.

Quality engineers, a term gaining popularity in agile organizations, have coding skills and work alongside developers to build and execute tests. These engineers ensure that quality is integrated into the development process, rather than being an afterthought. They focus on improving the overall testing process, reducing the time spent on identifying bugs, and increasing the quality of the final product. As testing becomes more automated, the need for quality engineers who can bridge the gap between development and testing grows.

Collaboration Between Developers and Testers

The relationship between developers and testers has become more collaborative in agile teams. Developers are increasingly responsible for writing unit tests to ensure the functionality of their code, while testers bring their expertise in spotting issues that developers may overlook. This collaboration is essential for delivering high-quality software quickly.

Testers must also understand the importance of communication. They need to be able to work closely with developers to identify and address issues early in the development process. By building mutual respect and understanding, developers and testers can collaborate more effectively, ensuring that the software is both functional and reliable.

Adapting to Agile Mindsets at the Stakeholder Level

Transitioning to an agile development process requires buy-in from stakeholders and executives. Kamat from Microsoft notes that stakeholders must understand that simply adding more resources or hardware won’t guarantee success. Agile principles emphasize flexibility, collaboration, and continuous improvement, which means that stakeholders must be willing to adapt their expectations and processes to accommodate these changes.

Stakeholders must also recognize the value of monitoring the application in production. As testing moves beyond the development cycle, real-time monitoring becomes crucial for identifying issues after the software is released. By supporting this shift, stakeholders can ensure that the software remains high-quality, even after it goes live.

Conclusion

Testing is not just a final step in the agile process; it is a crucial part of every stage of development. Agile cannot succeed without a strong, integrated testing strategy that includes automation, unit testing, static code analysis, and manual exploratory testing. By prioritizing testing early and continuously, businesses can ensure the success of their agile initiatives and deliver high-quality software at the speed required in today’s digital world. Whether through automated solutions, manual efforts, or independent test teams, businesses must make testing an ongoing part of their agile processes to remain competitive.

The software testing landscape has changed dramatically with the rise of agile methodologies. To stay relevant, testers, developers, and stakeholders must adapt to new roles, responsibilities, and strategies. By focusing on automation, collaboration, and real-time monitoring, teams can ensure the delivery of high-quality software that meets user needs and stands up to the challenges of the modern digital landscape. As agile practices continue to gain traction in global software development, including regions like Vietnam where outsourcing is growing, staying ahead of the curve will be crucial for organizations aiming for success.