Deploying Puppet code automatically with GitLab CI/CD pipelines

How to set up a basic pipeline to continually push code to Puppet

Photo showing a successful CI/CD deployment with a GitLab pipeline of Puppet code to a Puppet server, with output logs from the job


We’ll create a full pipeline for any push on master to deploy Puppet code to a production Puppet server instantly. We’ll need a special local user on the Puppet server, cut it a keypair and set up GitLab to do the rsync with a fresh container each time.

The environment

  • a Puppet server
  • a GitLab CE server
  • a bunch of clients that pull code at 3am

Puppet server config

  1. Add a user with useradd. I’ll use l-gitlabci for this.
  2. chown all of /etc/puppetlabs/code with l-gitlabci:puppet. This allows our user to write here, and the puppet software account to continue to run code.
  3. su into your new user’s account.
  4. Cut a key with ssh-keygen -t ecdsa (elliptic curves are faster than RSA)
  5. Get the base64 “version” of the private key with base64 -w0 and copy it to your clipboard.

GitLab server config

  1. Add a CI/CD variable in Settings > CI/CD > Variables called SSH_PRIVATE_KEY and paste in the base64 version from above. Protect and mask the variable.
  2. Set up a runner if you don’t have any. Runners are units of computing that execute pipeline jobs. For example a shell session on the GitLab server itself (not recommended) or a container that’s spun up and destroyed (recommended). You want to see a green dot before you proceed.
  3. Add a gitlab-ci.yml with just one stage (deploy). Breaking down the key steps:
    • we get a simple container going (e.g. alpine)
    • we add the openssh and rsync packages to the container
    • we test that ssh works
    • we go to the container’s root and make a .ssh directory
    • we decode our base64 SSH_PRIVATE_KEY variable back to normal and add it to our ssh environment
    • we turn off strict host key checking (discussion on this below)
    • finally we rsync our code
stages:          # List of stages for jobs, and their order of execution
  - deploy

deploy-job:      # This job runs in the deploy stage.
  stage: deploy
  environment: production
  image: alpine:latest
    - apk update && apk add openssh-client rsync
    - eval $(ssh-agent -s)
    - mkdir /root/.ssh
    - ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d)
    - echo "StrictHostKeyChecking no" >> ~/.ssh/config
    - echo "Deploying application..."
    - rsync -athv --delete ./
    - echo "Application successfully deployed."


Well… just push to master:

Not working for you?

  • Does /etc/ssh/sshd_config on the Puppet server allow PubkeyAuthentication?
  • Can you ssh -vvv l-gitlabci@puppetserver from a test server with a username with the same name, and the id_ecdsa key in ~/.ssh/?
  • Is your rsync command correct? -athv flags are more than enough. a implies r.
  • Did you copy the private key (correct) or the public key (incorrect) to GitLab?
  • Did you pipe the private key into the base64 encoder? (cat id_ecdsa | base64 -w0
  • Firewalls? DNS? Routing? SELinux?


When do the clients get the code?

Whenever they call puppet agent.

Don’t try engineer a pipeline to run the agent on the entire environment.

I’d suggest either:

  • all servers pull code at 3am local time
  • servers pull code based on a stability day (e.g. some servers are “Tuesday” servers, which pull on Tuesdays)

If a change is catastrophic, you haven’t killed all your infra and have some servers you can connect to for recovery.

Why no host key checking?

Yes, you could:

  • add an SSH_FINGERPRINT variable, or
  • curl a list of known host fingerprints from an internal web server

This marginally improves security, with a big downside risk. This is Puppet which is possibly managing all your infra.

In an emergency, faffing around with a host key issue when I’m trying to deploy to a new puppet server with a different host key is the last thing I want to have to remember to change. I actively discourage host key checking here.

GitLab environments?

A great idea would be to match Puppet environments with GitLab environments. I’d suggest having a branch for each environment. I’m not doing this yet.