Publish and Consume Private Terraform Modules in GitLab (Using CI_JOB_TOKEN)

If you're working in a real DevOps environment, chances are:

• Your Terraform modules are private

• Your infrastructure repos are private

• Different teams manage different groups

• Security policies don’t allow personal access tokens everywhere

Setup

Let’s say we have:

Group-1
 └── repo-b (Terraform module)

Group-2
 └── repo-a (repo consuming the Terraform module)

Both repos are:

• Private

• In different groups

• Using GitLab CI/CD

Goal:

• Publish module from repo-b

• Consume module in repo-a

• Authenticate using CI_JOB_TOKEN

• Use GitLab’s recommended Terraform module template

---

Publishing the Terraform Module (repo-b)

GitLab provides a built-in template for publishing Terraform modules to its registry.

That’s the cleanest and most production-ready approach.

Step 1: Name the repository properly

GitLab expects Terraform module projects to follow this naming format:

terraform-<module-name>-<provider>

Examples:

terraform-vpc-aws
terraform-ec2-aws
terraform-random-name-local

If we are building an AWS module, the suffix must match the provider.

Step 2: Add GitLab’s Terraform Module Template

Create .gitlab-ci.yml in repo-b:

include:
  - template: Terraform-Module.gitlab-ci.yml

variables:
  TERRAFORM_MODULE_SYSTEM: "aws"
  TERRAFORM_MODULE_VERSION: "${CI_COMMIT_TAG}"

Important:

• TERRAFORM_MODULE_SYSTEM is just the provider suffix (aws, local, random, azurerm, etc.)

• It does NOT configure the provider.

• It only affects the registry path.

That’s it. No curl. No manual packaging.

GitLab handles everything internally — including authentication via CI_JOB_TOKEN.

Step 3: Tag the Module

Terraform registry versions must follow semantic versioning.

git tag v0.1.0
git push origin v0.1.0

When we push a tag:

• GitLab packages the module

• Publishes it to the Terraform Module Registry

• Registers the version

We can verify it under:

Project → Operate → Terraform Modules

---

Allow Cross-Group Access

Since repo-a and repo-b are in different groups and private, GitLab blocks cross-project access by default.

We must explicitly allow it.

Enable Job Token Access (repo-b)

repo-b → Settings → CI/CD → Job token permissions

Enable:

Allow CI job tokens from other projects to access this project

Then add:

• repo-a project
  OR

• Group-2 (if we want all projects inside it to access)

This creates a trusted relationship.

Without this, we will encounter during terraform init:

401 Unauthorized

Consuming the Module (repo-a)

Now let’s use the module.

main.tf in repo-a:

module "vpc" {
  source  = "gitlab.com/group-1/terraform-vpc/aws"
  version = "1.0.0"
}

Notice the format:

gitlab.com/<group>/<project>/<provider>

NOTE: Not the full repo URL.

---

Configure CI Authentication (repo-a)

Inside .gitlab-ci.yml:

image:
  name: hashicorp/terraform:1.6
  entrypoint: [""]

variables:
  TF_TOKEN_gitlab_com: $CI_JOB_TOKEN

stages:
  - validate

validate:
  stage: validate
  script:
    - terraform init
    - terraform validate
    - terraform plan

Here’s the magic:

Terraform automatically reads the environment variable:

TF_TOKEN_gitlab_com: $CI_JOB_TOKEN

It uses that token to authenticate to GitLab’s registry.

Since:

• The pipeline generates CI_JOB_TOKEN automatically

• repo-b allows job token access

• The module is published in the registry

Terraform can securely download the module.

---

Why CI_JOB_TOKEN Is Better Than PAT

Using Personal Access Tokens:

• Requires manual creation

• Can expire

• Must be stored as secrets

Using CI_JOB_TOKEN:

• Automatically generated per job

• Scoped to project

• Rotated automatically

• Safer and cleaner

This is how enterprise pipelines are designed.

---

Common Errors and Fixes

Terraform has no command named "sh"

Cause: Terraform Docker image has an ENTRYPOINT.

Fix:

image:
  name: hashicorp/terraform:latest
  entrypoint: [""]

---

Final complete flow