Orchestrating CI/CD Pipelines for Microservices at Scale

The transition from a monolithic architecture to microservices is often celebrated as a major win for development agility. However, from an ITOps and infrastructure perspective, it introduces a significant challenge: pipeline sprawl.

What used to be a single, easily manageable CI/CD pipeline for a monorepo suddenly multiplies. In our case, we found ourselves facing the daunting task of managing 15 distinct pipelines.

Here is how we tackled the governance and maintenance nightmare that comes with microservice CI/CD orchestration.

The Problem: The Governance Nightmare of Distributed Pipelines

When you split a monolith into 15 microservices, giving each repository its own independent Jenkinsfile, you inevitably run into severe IT governance and compliance issues:

1. Maintenance Overhead: If a security requirement changes or a new deployment step is added, that change must be manually propagated across 15 different repositories and integrated into every active developer branch.
2. Security & Compliance Risks: In environments where developers are not strictly DevOps experts, having local control over the pipeline means they can easily alter or bypass critical security checks (like SAST or SonarQube) if those checks block their deployment.
3. Loss of Integrity: Applying the exact same enterprise directives across the board becomes nearly impossible in a frequently changing IT environment.

The "Before" Architecture

┌────────────────┐      ┌─────────────────┐      ┌─────────────────────────┐
│ Microservice A │ ───> │ Local Pipeline  │ ───> │ Build ─> Test ─> Deploy │
└────────────────┘      └─────────────────┘      └─────────────────────────┘
┌────────────────┐      ┌─────────────────┐      ┌─────────────────────────┐
│ Microservice B │ ───> │ Local Pipeline  │ ───> │ Build ─> Test ─> Deploy │
└────────────────┘      └─────────────────┘      └─────────────────────────┘
      ...                       ...                          ...
┌────────────────┐      ┌─────────────────┐      ┌─────────────────────────┐
│ Microservice O │ ───> │ Local Pipeline  │ ───> │ Build ─> Test ─> Deploy │
└────────────────┘      └─────────────────┘      └─────────────────────────┘

Result: High maintenance, low standardization, and zero centralized control.

The Solution: The Master Orchestrator Pipeline

To regain control, we shifted from a decentralized pipeline model to a Centralized Orchestration Model.

Using Jenkins' ability to trigger external pipelines, we created a single, highly robust Master Pipeline. Now, the Jenkinsfile residing in each microservice repository contains only a single call to this master orchestrator, passing its specific variables as parameters.

The "After" Architecture

┌────────────────┐
│ Microservice A │ ───┐
└────────────────┘    │
┌────────────────┐    │   ┌─────────────────────────┐   ┌─────────────────────────┐
│ Microservice B │ ───┼─> │ MASTER JENKINS PIPELINE │ ─>│ Build ─> SAST ─> Deploy │
└────────────────┘    │   └─────────────────────────┘   └─────────────────────────┘
      ...             │
┌────────────────┐    │
│ Microservice O │ ───┘
└────────────────┘

Result: Write once, run everywhere. One single source of truth for CI/CD.

Handling Exceptions: The YAML + CODEOWNERS Trick

Centralization is great, but reality is rarely perfectly standardized. What happens when the project must adhere to strict SAST (Static Application Security Testing) compliance, but a specific legacy microservice isn't mature enough to pass those checks yet? If the master pipeline enforces it, the build will fail continuously.

To handle these exceptions without hardcoding bypasses into the pipeline, we introduced a declarative configuration file: pipeline-config.yaml, located in each repository.

This file maps out the steps of the master pipeline, allowing toggles (true/false) for specific stages:

# pipeline-config.yaml
pipeline_stages:
  unit_tests: true
  linting: true
  sast_analysis: false # Temporarily disabled until tech debt is resolved
  deploy_to_dev: true

Enforcing Integrity

If developers can just change sast_analysis to false, haven't we lost our governance? This is where GitHub's CODEOWNERS feature saves the day.

We added the pipeline-config.yaml file to the .github/CODEOWNERS file, assigning approval rights strictly to the ITOps/DevOps team.

# .github/CODEOWNERS
pipeline-config.yaml  @drefactor-itops-team

Because of how CODEOWNERS works, a Pull Request modifying the pipeline configuration can never be merged without explicit approval from the designated owners. Furthermore, it triggers an automated email to the ITOps team the moment a developer attempts to bypass a security check.

The Rollout Strategy

Once the centralized architecture and the YAML safeguards are in place, rolling out compliance becomes a manageable, Agile process:

1. Onboard: Connect all microservices to the Master Pipeline with strict checks turned false in their YAMLs.
2. Prioritize: Follow your framework (Scrum, SAFe) to create backlog items for resolving tech debt.
3. Enforce: Gradually switch restrictions (Linters, Code Smells, SAST) to true category by category, microservice by microservice, as they reach maturity.

By combining a master pipeline with code-owned YAML configurations, we achieved the perfect balance: deep standardization for the infrastructure team and frictionless deployments for the developers.