This is a small story about a shell feature that broke the CI.

Background

A development team reached out to me and asked for help, as terraform consistently crashed for them:

What are we doing?

During the GitLab CI run, the following simple shell code is executed to put a Service Account Key from a CI variable into a file, which is later read by Terraform to authenticate against the Google Cloud APIs.

1
2
3
4
5

.terraform_template:
  before_script:
    - echo $SERVICE_ACCOUNT_CONTENT > service_account.json
    - export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/service_account.json
    - terraform init

This template has worked fine for a very long time. So why is Terraform now complaining just for this project, that the file defined in GOOGLE_APPLICATION_CREDENTIALS is invalid JSON?

What is happening, and why?

Let's take a look the content of $SERVICE_ACCOUNT_CONTENT (redacted):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
    "type": "service_account",
    "project_id": "gcp-project",
    "private_key_id": "",
    "private_key": "-----BEGIN PRIVATE KEY-----\n...\n...\n...\n-----END PRIVATE KEY-----",
    "client_email": "sa@gcp-project.iam.gserviceaccount.com",
    "client_id": "...",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "..."
}

Looks like normal JSON Service Account Key for Google Cloud, right?

I copied the CI variable content into a local file and then ran some commands to test something (check out jq if you don't know it already):

$ SERVICE_ACCOUNT_CONTENT=$(cat account.json)
$ echo $SERVICE_ACCOUNT_CONTENT | jq
parse error: Invalid string: control characters from U+0000 through
U+001F must be escaped at line 22, column 1

Huh? 🤨

How does shell variable expansion work?

When you put a variable into a command, the variable is replaced exactly with the content of it before being passed to the application.

VAR="hello world"

# The command:
echo $VAR

# Becomes:
echo hello world

# Outputs:
hello world

That means if your variable contains a space, you'll have multiple arguments to the application. If your variable is a path that contains a space, it just breaks. Especially macOS users love to fall into this trap:

FILENAME="/Users/mitch/Library/Application Support/something.txt"

# The command:
cat $FILENAME

# Becomes:
cat /Users/mitch/Library/Application Support/something.txt

# Outputs:
cat: /Users/mitch/Library/Application: No such file
cat: Support/something.txt: No such file

Additionally, certain escape sequences are also supported.

In our JSON case, we have multiple \n (newlines) inside a string (the private key), which is being processed by the shell as well:

CONTENT='a\n\b\nc' # If we use ', no interpolation is done

# The command:
echo $CONTENT

# Becomes:
echo a\nb\nc

# Outputs:
a
b
c

After the JSON is (accidentaly) processed by bash and put through echo, there's now "real" line breaks within strings - which is disallowed by JSON.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
    "type": "service_account",
    "project_id": "gcp-project",
    "private_key_id": "",
    "private_key": "-----BEGIN PRIVATE KEY-----
  ...
  ...
  ...
  ...
  -----END PRIVATE KEY-----",
    "client_email": "sa@gcp-project.iam.gserviceaccount.com",
    "client_id": "...",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "..."
}

What's the fix?

In general, always enclose shell variables with quotation marks (") when passing them to applications. This ensures they're passed as single argument and won't get split up.

And because we're using YAML for the CI definition, we also need a different syntax to define the command strings:

1
2
3
4

.terraform_template:
  before_script:
    - |
      echo "$SERVICE_ACCOUNT_CONTENT" > service_account.json

Finding shell issues more quickly

To quickly find issues like these in your bash scripts, use the wonderful shellcheck.

How to prevent that in the first place?

Alternatively, GitLab CI is also able to write variables directly to files and puts the filename into the variables. That way, you won't even have to bother with issues like that.