The Joy of Hex

Drunken Monkey Coding Style with a hint of Carlin humor

Jun 24, 2018 - 8 minute read - docker development devops

The rabbit hole is deep when trying to switch from environment variables file to Docker secrets

We’ve been using docker for development for a while, as at this point it is easier to manage than Vagrant/Ansible even with the performance penalty on MacOSX.

As we have multiple containers setup, preferred way to share credentials and config was to use environment variables which were stored in .env file, and to keep our sanity we’d use docker-compose, then the project would be deployed to servers and the cycle would repeat. This time the deployment would be dockerized, so our project should be able to run docker swarm (let us not go into discussion over k8s or docker swarm).

Naively I tried to deploy containers to the stack and learned that reading .env files does not work on docker swarm, yes, you could run source .env && docker stack deploy .. but that is kind of not the point, I’d like not to think about those things. Preferred way to handle the credentials and configuration is to use Docker Secrets. And of course, you create secret one by one, because why not… Add to that the following post, and I’ve seen the error of my ways.

There is an .env file that has some fourtyish entries for env vars that are to be converted to secrets, there is no way in this universe or the next that I am going to do that by hand. So I came up with the following bash helper

convert-env-to-secrets.sh 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#!/usr/bin/env bash

ARGS="$@"

ARGS="${ARGS:- .env}"

SECRET_APPEND="secrets:\n"

while read p; do
    # eliminate any lines that are empty, or start with # (comments)
    if [[ $p == \#* ]] || [[ -z "${p// }" ]]; then
        continue
    fi

    # split the line by =
    IFS='=' read -r NAME VALUE <<< "$p"

    docker secret rm $NAME
    printf "${VALUE}"|tr -d '\n' | docker secret create $NAME -

    SECRET_APPEND="${SECRET_APPEND}  - ${NAME}\n"

done <$ARGS

echo "add following to your service that uses the secrets:"
echo -e "${SECRET_APPEND}"

Not exactly the nicest thing, but it gets the job done, if you don’t specify the filename to read, it defaults to the .env. Keep in mind that Docker swarm must be running when you run this script or you will get a nice error like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

Provided swarm was running, you should now have the secrets defined, which is awesome. At the end, the script also prints out all your defined secret names so you can append them to the services that will be using them.

You can verify that your secrets have been generated by executing docker secret ls

Now to update your docker-compose.yml so it can use secrets as well, and you have unified approach.

Suppose we have the following docker-compose file

docker-compose.yml 
1
2
3
4
5
6
7
8

version: '3.6'
services:
  web_app_cli:
    image: code4hire/dev-images:php-7.2-cli
    hostname: "web_app_cli"
    working_dir: /application
    volumes:
      - ./web/application:/application

To setup the secrets, you need two things:

  1. define a top level secrets to list your secrets
  2. on the service that is going to consume that secret, you need secrets with names enumerated
docker-compose.yml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

version: '3.6'
services:
  web_app_cli:
    image: code4hire/dev-images:php-7.2-cli
    hostname: "web_app_cli"
    working_dir: /application
    volumes:
      - ./web/application:/application
    secrets:
      - my_secret
      - my_other_secret

secrets:
  my_secret:
    external: true      
  my_other_secret:
    external: true

Well, that was easy, wasn’t it… When you run your service with docker-compose up you get a nice surprise

1
2

WARNING: Service "web_app_cli" uses secret "my_secret" which is external. External secrets are not available to containers created by docker-compose.
WARNING: Service "web_app_cli" uses secret "my_other_secret" which is external. External secrets are not available to containers created by docker-compose.

OK, so now you know that that approach does not work, and that docker-compose can’t use secrets directly, but there is an alternative, you can specify the file that holds the secret value to be used. But that means that you should now split your .env file into separate files that have the name of key and have the contents of value

convert-env-to-secret-files.sh 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#!/usr/bin/env bash

ARGS="$@"

ARGS="${ARGS:- .env}"

SECRET_APPEND="secrets:\n"

while read p; do
    # eliminate any lines that are empty, or start with # (comments)
    if [[ $p == \#* ]] || [[ -z "${p// }" ]]; then
        continue
    fi

    # split the line by =
    IFS='=' read -r NAME VALUE <<< "$p"

    DIRECTORY="secrets"
    FULL_PATH="${DIRECTORY}/${NAME}"

    [ -d ${DIRECTORY} ] || mkdir -p ${DIRECTORY}
    echo "${VALUE}"|tr -d '\n' > ${FULL_PATH}

    SECRET_APPEND="${SECRET_APPEND}  - ${NAME}\n"

done <$ARGS

echo "mount the secrets in service that uses them by adding - ./secrets:/run/secrets under volumes"
echo "add following to your service that uses the secrets:"
echo -e "${SECRET_APPEND}"

And then you modify the docker-compose.yml to use the files for secrets…

docker-compose.yml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

version: '3.6'
services:
  web_app_cli:
    image: code4hire/dev-images:php-7.2-cli
    hostname: "web_app_cli"
    working_dir: /application
    volumes:
      - ./web/application:/application
    secrets:
      - my_secret
      - my_other_secret

secrets:
  my_secret:
    file: ./secrets/my_secret.txt
  my_other_secret:
    file: ./secrets/my_other_secret.txt

So awesome that should work… In principle yes, until you try to mount 10+ secrets on MacOS then you get the following error

1

Error response from daemon: Mounts denied: write unix ->osxfs.sock: write: broken pipe

OK, to be fair, this is MacOS specific, but still… If you paid attention to the contents of the bash script, you would have noticed I already solved that… By cheating.

Docker secrets are mounted as read only volumes in your container, no one said where they come from, so you modify the docker-compose.yml

docker-compose.yml 
1
2
3
4
5
6
7
8
9

version: '3.6'
services:
  web_app_cli:
    image: code4hire/dev-images:php-7.2-cli
    hostname: "web_app_cli"
    working_dir: /application
    volumes:
      - ./web/application:/application
      - ./secrets:/run/secrets

There, your docker-compose can now use the secrets, you application can now consume them, and you have a unified setup for both local dev and production.

At this point I’m considering switching to HashiCorp Vault or something similar, as Docker secrets look like a potential source of never ending woes