Pipeline Template

24 Jan, 2025

I am using GitLab CI/CD for all my DevOps automations. Starting from simple scripts that are executed yearly, monthly, etc... ending on the complex deployment scenarios for the all the Artifactory instances I am managing.

GitLab CI/CD pipelines are defined in YAML file. For simple pipelines, I can write it using just this one file. For more complex solution, I can import some chunks from other files in the same repository, as well as from files stored in other project's repositories.

Pipeline jobs written in YAML can reuse code by using YAML Anchors, !reference tags or extends keyword. While I am not a big fan of the first two, seeing them as harder in maintenance, I was broadly using the extends keyword. Until I found include with spec:inputs parameters.

Pipeline

I am managing multiple Artifactory instances with a Terraform, all storing their resources' configuration in one repository. The pipeline has the following stages: Validate, Test, Download, Build, and Deploy.

Validate is running terraform fmt and terraform validate.
Test is running multiple tests on different types of resources, e.g. checking if the users accounts used in Permission Targets do exist on the target instance.
Download is used to download dedicated Terraform modules and other required resources.
Build is executing terraform plan, and
Deploy, that runs only after merge to main, is executing terraform apply.

Before using `spec:inputs`

So without the templates using inputs, I had the following pipeline definition (simplified):

# .gitlab-ci.yml

include:
  - template: Terraform/Base.gitlab-ci.yml

stages:
  - build
  - deploy

# job-template
.build_job_template:
  extends:
    - .terraform:build
  variables:
    TF_ROOT:
    STATE_NAME:
  script:
    - echo "This is my build job"

.deploy_job_template:
  extends:
    - .terraform:deploy
  variables:
    TF_ROOT:
    STATE_NAME:
  script:
    - echo "This is my deploy job"

# conditions
.if_merge_request: &if_merge_request
  if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

.if_default_branch: &if_default_branch
  if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

# patterns
.test_pattern: &test_pattern
  - "instances/test-instance/**/*.tf"

# rules
.build_rule_test:
  rules:
    - <<: *if_merge_request
      changes: *test_pattern
    - <<: *if_default_branch
      changes: *test_pattern

.apply_rule_test:
  rules:
    - <<: *if_default_branch
      changes: *test_pattern
      when: manual

# test instance jobs
test_instance_build:
  extends: .build_job_template
  variables:
    TF_ROOT: instances/test-instance
    STATE_NAME: test
  environment:
    name: test
    url: https://test-instance.com/
    action: access

test_instance_deploy:
  extends: .deploy_job_template
  variables:
    TF_ROOT: instances/test-instance
    STATE_NAME: test
  dependencies:
    - test_instance_build
  environment:
    name: prod
    url: https://prod-instance.com/
    action: access

This is the definition of the pipeline, working on one instance. It starts a build job on every commit inside a merge request. If MR is merged to main, it will run the build job, and wait for a manual run of the deploy job. Pretty standard.

The problem starts when there is a need to add an additional instance. To do this, I need to duplicate a lot of code, with some minor differences only:

create two new jobs for build and deploy to new instance - 20 lines of code
create new pattern for new instance - 2 lines of code
create two new rules for new instance - 11 lines of code

This gives 33 lines of code just to add a new instance! That's insane. And the code above is missing jobs responsible for running test cases, and those have dedicated rules and patterns as well.

There are also the rules defined in three different places. If someone likes to read the pipeline code, to understand it, it is required to constantly scroll it up and down to check what the build_rule_test does and what is the scope of test_pattern.

With `spec:inputs`

Let's now compare the above with the same pipeline, but created with template using inputs variables. The template structure is as follows:

spec:
  inputs:
    variable1:
      description: Some description
      type: TYPE

---

# jobs definition

It is important to add the spec section before the ---. More in the documentation.

My inputs section needs the following input variables defined in the spec section: environment, job_suffix, state_name. Next, after the ---, I have all the job templates that will be used by all the instances. Each job has the environment suffix in the name, so build job for test environment has different name than build job for other instances:

# .gitlab/pipeline-template.yml

spec:
  inputs:
    environment:
      description: "Environment name, same as the path in instnaces directory."
      type: string
    job_suffix:
      description: "Short name of the environment."
      type: string
    tf_state_name:
      description: "Name of the terraform state in http backend"
      type: string

---

include:
  - template: Terraform/Base.gitlab-ci.yml

stages:
  - build
  - deploy

build:$[[ inputs.job_suffix ]]:
  stage: build
  extends:
    - .terraform:build
  variables:
    TF_ROOT: $CI_PROJECT_DIR/instances/$[[ inputs.environment ]]
    STATE_NAME: $[[ inputs.tf_state_name ]]
  script:
    - echo "This is my build job"
  rules:
    # run in merge request
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - instances/$[[ inputs.environment ]]/**/*.tf
    # run on main
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - instances/$[[ inputs.environment ]]/**/*.tf
  environment:
    name: $[[ inputs.environment ]]
    url: https://$[[ inputs.environment ]]
    action: access

deploy:$[[ inputs.job_suffix ]]:
  stage: deploy
  extends:
    - .terraform:deploy
  variables:
    ENVIRONMENT_NAME: $[[ inputs.environment ]]
    TF_ROOT: $CI_PROJECT_DIR/instances/$[[ inputs.environment ]]
    TF_STATE_NAME: $[[ inputs.tf_state_name ]]
  dependencies:
    - build:$[[ inputs.job_suffix ]]
  rules:
    # hide in merge request
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - instances/$[[ inputs.environment ]]/**/*.tf
      when: never
    # run on main, but trigger manually
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - instances/$[[ inputs.environment ]]/**/*.tf
      when: manual
  environment:
    name: $ENVIRONMENT_NAME
    url: https://$[[ inputs.environment ]]

I have removed YAML anchors for pattern, conditions and rules definitions and moved them to jobs. This way, it is easy to understand when the job will be executed. No need to scroll or check in other files. There is a small amount of repetition, but it is easier to maintain.

But, is there any other benefit of the code in compare to previews version. Oh, YES.

Let's now use this code to create pipeline for test env:

# .gitlab-ci.yml
---

- local: ".gitlab/artifactory_template.yml"
  inputs:
    environment: "test-instance.com"
    job_suffix: "test"
    tf_state_name: "test"

That is all. It's only 5 lines of code to add a new instance. No need to create dedicated jobs, just call the template with correct input variables. With the above instance added, the pipeline will create the following jobs:

test:build
test:deploy

We can add new instance as follows:

# .gitlab-ci.yml
---

- local: ".gitlab/artifactory_template.yml"
  inputs:
    environment: "test-instance.com"
    job_suffix: "test"
    tf_state_name: "test"

- local: ".gitlab/artifactory_template.yml"
  inputs:
    environment: "prod-instance.com"
    job_suffix: "prod"
    tf_state_name: "prod"

With this, the following jobs will be created:

test:build
test:deploy
prod:build
prod:deploy

If this is not great, then what is?

#work

Pipeline Template

Pipeline

Before using spec:inputs

With spec:inputs

Before using `spec:inputs`

With `spec:inputs`