This is a small story about a shell feature that broke the CI.
A development team reached out to me and asked for help, as terraform consistently crashed for them:
What are we doing?
During the GitLab CI run, the following simple shell code is executed to put a Service Account Key from a CI variable into a file, which is later read by Terraform to authenticate against the Google Cloud APIs.
This template has worked fine for a very long time. So why is Terraform now complaining just for this project, that the file defined in
GOOGLE_APPLICATION_CREDENTIALS is invalid JSON?
What is happening, and why?
Let's take a look the content of
Looks like normal JSON Service Account Key for Google Cloud, right?
I copied the CI variable content into a local file and then ran some commands to test something (check out jq if you don't know it already):
How does shell variable expansion work?
When you put a variable into a command, the variable is replaced exactly with the content of it before being passed to the application.
That means if your variable contains a space, you'll have multiple arguments to the application. If your variable is a path that contains a space, it just breaks. Especially macOS users love to fall into this trap:
Additionally, certain escape sequences are also supported.
In our JSON case, we have multiple
\n (newlines) inside a string (the private key), which is being processed by the shell as well:
After the JSON is (accidentaly) processed by
bash and put through
echo, there's now "real" line breaks within strings - which is disallowed by JSON.
What's the fix?
In general, always enclose shell variables with quotation marks (
") when passing them to applications. This ensures they're passed as single argument and won't get split up.
And because we're using YAML for the CI definition, we also need a different syntax to define the command strings:
Finding shell issues more quickly
To quickly find issues like these in your bash scripts, use the wonderful shellcheck.
How to prevent that in the first place?
Alternatively, GitLab CI is also able to write variables directly to files and puts the filename into the variables. That way, you won't even have to bother with issues like that.