A DevSecOps transformation story

Our IT team’s successful migration of hundreds of employee facing apps to a cloud-native environment highlights how businesses can realize cost savings and productivity gains without compromising on security.

DevSecOps is crucial to any organization’s digital transformation effort. It has wide ranging benefits such as improved productivity and security, cost savings and faster time to market. As with any transformation effort, it comes with its set of challenges, including infrastructure, tool sprawl, the need to balance speed and security and generating buy-in within the team as well as leadership. Our IT team supports hundreds of applications with thousands of components, making DevSecOps essential for seamless, reliable service delivery.

Most of Cognizant’s app ecosystem uses a mobile-first, cloud-first strategy on modern platforms. However, the company’s engineering processes, and security were built in the pre-cloud era. As the existing DevSecOps tools were unable to meet our digital transformation needs, we decided to reshape our core processes and tools, aligning them with the digital strategy of cloud-native development.

We had 270+ applications, 20,000+ components, and 7490 pipelines across 2,170 repositories that needed to be migrated from legacy to a modern DevSecOps stack within just six months.

We faced several initial challenges such as:

• Siloed, opaque deployments: Dev, Ops and Security teams working separately meant that we faced delays in test execution. Deployment was done using in-house tools without support for cloud and mobile technologies. We also lacked end-to-end code traceability from requirements to deployment.
  
  
• Avoidable delays: Differences between customer expectations and delivered features during the user acceptance testing (UAT) delayed production and caused additional friction. Delivery timelines failed to consider scoping requirements and team capacity.
  
  
• Legacy tech and processes: We were dependent on legacy source code version control tools that were not cloud ready and had to work with large and ambiguous business requirement documents. Key metrics such as lead and process time had to be extracted from various tools.
  
  
• Poor collaboration: Working extensively through email meant that information traceability had become difficult. Lack of transparency and information flow between IT and business also created a low-trust relationship, leading to unsustainable workloads.
  
  
• Reactive security: Application security was always an afterthought and security scans were ad hoc and manual.

A measured approach to automated DevSecOps

We knew that this transformation required us to create a unified approach and language that encouraged business stakeholders to work together. We began with small releases to foster collaboration between business and IT, leveraging automation through continuous integration and continuous delivery (CI/CD), and embedding security into each iteration as a core cultural practice.

Making changes to 7,490 pipelines individually would be a daunting task, so the team created a templatized model that enabled the deployment logic to be implemented centrally. Next, we aligned the DevSecOps tools (SAST, DAST & OSS) to enterprise architecture and corporate security guidelines and standards. In doing so, we implemented a first-of-its-kind automated, authenticated DAST scan for 150+ URLs.

To effectively use the security tools and their licenses and ensure seamless execution, we developed proofs of concepts (PoCs) to explore CAST, SAST, DAST and OSS tools, configuring them to meet our needs and integrating them into the DevOps pipeline. After setting up the tools, we ran a pilot with sample app teams to validate operational efficiency and integration, incorporating feedback into the final execution.

Enabling DevSecOps scans for 200+ apps in an automated DevOps pipeline involved more than 1000 developers and was initially estimated to take six months. However, the teams conducted a series of workshops for batches of apps, completing the scans and configurations for all the pipelines in just one month.

We then transitioned from lengthy and time-consuming manual and ad hoc assessment of CAST, SAST, DAST and OSS scans to automated and integrated scans and conducted remediation validation checks within the Azure DevOps (ADO) CI/CD pipeline.

Today, only secure code is deployed into production. Management has increased visibility into security risks, allowing us to make informed risk-based assessment and remediation decisions, leading to improved compliance and security.

The biggest challenge we faced was to foster the adoption of a security mindset within the teams. To this end, we adopted a unique approach organizing large-scale, hands-on workshops, and creating “Show and Tell” success stories. We published weekly and monthly DevSecOps score for the stakeholders and engaged with the teams through regular “Community of Practice” sessions and a Yammer community to share best practices. By automating project setups and repo creation using a factory-based model and focused workshops, we completed the migration in half the originally planned time.

We observed an 85% improvement in lead time, 85%-90% in process time, and a 90% reduction in security risks.

Our digital transformation approach

Our transformation was driven by a multifaceted approach focusing on technical as well as cultural aspects. Here’s how we did it:

1. Prioritizing training: We mandated annual secure software development lifecycle (SSDLC) and DevSecOps role and skill up training for the 2500+ member team.
   
   
2. Enabling transparency and traceability: Business product owners are engaged with IT teams to build trust and transparency. We ensure end-to-end traceability, visibility into code changes and better management of requirements using agile life cycle management tools.
   
   
3. Creating a collaborative environment: The dev and operations worked as one team with security champions partnering in the process. We also practice iterative delivery, with teams operating within capacity and conducting mid-sprint demos to get quick feedback, thus reducing rework and improving functional clarity.
   
   
4. Embracing embedded security practices and reviews: Security practices, reviews and signoff tollgates are embedded into all phases of the development life cycle to reduce risk and deploy secure code.
   
   
5. Adopting new strategies and model: We deployed a branching strategy and used the Agile SSDLC Maturity Model to drive the digital DevSecOps culture.
   
   
6. Sharing metrics: Key DevSecOps metrics such as static and dynamic application security testing (SAST/DAST) and open-source software (OSS) compliance scores are shared with leadership team to enhance app security through risk-based assessment.

Results

Source: Cognizant
Figure 1

This transformation effort created significant cost savings for the organization (Figure 1). It had a massive impact on productivity in the form of 288,000 hours saved per annum. The key areas where these savings were achieved were DAST scan/release, SAST scan/release and pipeline changes. Going forward we expect this effort to realize returns worth $34.5 million over the next five years. Other key outcomes were as follows:

• Process and tools onboarding: 93% (200+) apps onboarded
• Security controls: 25,000 controls implemented with 96% assessed and certified
• Security test automation: 100% SAST and OSS automated scans; 40% DAST scans
• Risk reduction: 50% reduction in vulnerabilities; 100% compliance from external ISO audit
• Security culture: 70 hours of DevSecOps community of practice connects for 200 developers
• Time saved through automation: 2 days to 5 months per SAST scan and 15 days to 2 months per DAST scan

Through this process, we drove significant organizational change. Our application security posture has improved, our workflows are cleaner, teams are more engaged, we’ve witnessed a marked improvement in our code quality and there is higher commitment and reliability for IT and business.

Looking forward

Adapting to new changes is the key to any modernization journey. As we prepare for the road ahead, we believe this experience has provided us with valuable insights into how we can tackle the challenges that lie ahead. We are well-placed to deliver secure, faster services, powered by end-to-end automated DevSecOps with minimal touch points.