Rethinking Testing: You Can’t Have CI/CD Without CT


While CI/CD is the objective for a growing number of organizations, accounting for the holistic process involved in accomplishing that is required. This includes the need for continuous test to be achieved across the lab to live process. Learn the solution components to achieve that.

Everyone’s talking about it. You many know it as CI/CD (continuous integration / continuous delivery [or deployment]), DevOps, NetDevOps or SecDevOps, but it all relates to the Telecom Innovation Pipeline organizations are adopting to bring their products and services to market faster, and with assurance of high quality to remain competitive in the market. Achieving CI/CD is near the top of the priority list for a majority of organizations today and it represents a defining factor in competitive advantage.

CT’s context in the telecom innovation pipeline

While CI/CD is the objective, it’s also a process, and a process is only as strong as its weakest link. While continuous testing (CT) is called out in the diagram below, what the graphic doesn’t show is that CT is actually integrated into in every step of the process. And a failure in CT anywhere in the process chain means CI/CD cannot be assured. While optimal CI/CD is the goal for all adopting organizations, how it is achieved means addressing the devils in the details. So how do organizations realistically achieve CT?

Accounting for the J-Curve effect

From the start, even in the planning stage, expectations within an adopting organization must be set with the stakeholders. Some leaders in adopting organization may anticipate results quickly (they certainly want them!). Those expectations, however, must be put into a proper perspective with an understanding of the J-Curve effect – where even if everything goes as planned – there is a marked (yet temporary) loss in productivity before the benefits of optimized lab management and test automation, driven by DevOps efficiencies, can be achieved.

The reason for this is that there is a significant reshaping of the entire test lab infrastructure around testing and assurance, which is no small undertaking (see the blog on the phases of evolution in Optimal Lab as a Service Automation). Think of it as something like turning an oil tanker in the middle of the ocean. While the decision is made by the captain, and conveyed in less than a minute, the actual execution of the navigation can take a half hour to readjust course, or more, based on the conditions at sea and the extremity of the course adjustment.

The same is true with lab optimization as well as test and assurance automation. But unlike open sea navigation, organizations have a secret weapon: they can bring in an expert partner with expertise in all the areas of physical and virtual lab management, test campaign automation, and DevOps best practices, that puts them on track quickly, and with assurance. More on that approach later.

Breaking CT down

From the beginning, stakeholders must recognize the challenges inherent in adopting CI/CD. What this means is that independent of the J-Curve Effect, there can be elements within an organization that, if unresolved, can lead to a solution adoption’s failure. These all boil down to the people, processes and technology, the essential elements of a solution of this nature, all of which influence each other continuously. A sampling of potential challenge points includes:

  • People. Testers from development often feel QA tests are unreliable. Most QA engineers building the testing automation are not experienced software developers and don’t know methods to make test code scalable for future requirements. Worst case is they fall back to manual ad-hoc testing (non-repeatable). Meanwhile, software engineers are working on technology trends and priorities QA engineers are unaware of. Such siloed and misaligned endeavors in testing too often result in time-consuming and manual corrective work that cripples the advantages of CT.

  • Processes. CT doesn’t happen overnight, and once achieved, the technology supporting it must be maintained. When budgets get tight, this is often one of the first things to go. When it does, CT is hindered, impacting the entire CI/CD process. Long-term planning, proper allocation of budget and resources for sustained maintenance of the solution, must be accounted for to ensure the success of ongoing optimal CT capabilities.

  • Technology. A common occurrence is a well-intentioned DIY (do it yourself) lab and test automation solution created for a group within a larger organization, QA for example, to address specific test automation objectives. Yet, when the solution is required to expand its capabilities to accommodate more stakeholders in the organization, who have different requirements, the solution can’t scale. The result: achieving CT organization-wide is not possible.

Adopting the right strategic approach to achieve CT

Whether integrating a full-scale adoption, or choosing a lab and test automation partner with a managed solution to achieve your testing goals, having an understanding of the framework and the required components of success is advantageous. An overview of the requirements most organizations seek in their solution components generally target comprehensive advantages in testing and assurance.

The foundation of success for CT involves three solution areas, all of which interact seamlessly:

1. Lab automation fundamentals

Lab as a Service (LaaS) solutions must provide the ability to consolidate and automate labs and testbeds across regions, as well as optimize lab workflows through automation. It must be able to schedule the use of shared resources, rapidly model and instantiate test environments, establish lab policies, automate test execution, and deliver analytical visibility of results and resource utilization – all through a web interface. LaaS should be highly scalable with an enterprise design enabling fast adoption and ROI.

2. Test automation objectives

Tightly integrated with LaaS, a Test as a Service (TaaS) solution should be able to leverage the broadest set of domain tests possible, spanning the lifecycle. It must also streamline collaboration between developers and QA, as well as providers and vendors. It must be intelligent with environment-aware test case management, execution, and analysis, and be easy to use, so tests can be published, scheduled, and shared with anyone, anywhere. A TaaS solution must provide a system that will intelligently deploy tests to minimize test times and streamline test execution from lab to production environments. The TaaS platform should empower users to create, automate and scale testing, test cases and, campaigns to address all these needs. 24/7 testing, from anywhere.

3. The DevOps imperative

Once the physical infrastructure is consolidated, centralized and virtualized for maximized efficiencies via LaaS, with a scalable TaaS automation solution, the next phase of advancement to achieve CT involves leveraging DevOps and Agile principles. These employ improved data management and automation practices that build quality into the solution by integrating test labs with development and operations (DevOps). This creates easily distributed knowledge through reproducible environments. DevOps practices deliver increased productivity through acceleration of integrated dev-QA workflows. Essentially, DevOps brings it all together to optimize testing and make CT possible.


DevOps brings it all together to optimize testing and make CT possible.

The breadth and depth of proficiencies required to realize enduring CT

Throughout the lifecycle of the solution and across its entire landscape, from LaaS, TaaS and DevOps, the solution must remain responsive to emerging requirements. Some organizations, however, don’t have the time or budget to get the infrastructure and expertise they need to validate on the scale required. That’s why some rethink their approach to testing and turn to solution partners with that expertise. These partners offer turnkey test cases from a broad library and deliver the knowledge of how to build a collaborative environment so seamless work across internal teams, as well as service providers and vendors, can be achieved quickly and without error.

The test and lab automation solution partners also should have deep expertise in all the emerging technologies such as: 5G, 5G Core, cloud infrastructure, secure SD-WAN, SDN, NFV, Open RAN, security, Wi-Fi6/E, and more. They should be able to provide advanced test automation within the framework of LaaS and TaaS, empowered by DevOps best practices, delivering state-of-the-art testing. As well, their approach should offer a next-gen technology platform which addresses the requirements of the 21st century test lab and supports service-oriented architecture. Those organizations that recognize the advantage of a solution partner filling the gaps in their capabilities can realize a defining factor in gaining a competitive advantage in the market.

Read the white paper The Criticality of Continuous Test for CI/CD Success右箭头图标

Learn more about Spirent’s Managed Solutions右箭头图标




Bob Dietrich


拥有30年从业经验的Bob Dietrich是一位成就卓著的电信和云解决方案架构师。他专注于OSS和网络管理,在各种类型解决方案的整个生命周期提供支持,包括从愿景、商业案例论证、设计、测试、部署到客户接受度和满意度等各个方面。