Secure your Docker passwords with Hashicorp Vault and Ansible (Part 2: Automate Docker startup with Ansible)
This is the second part of a two-part series on securing Docker passwords. Using Authelia as an example, I show how to automate the container start.
If you already have Authelia you can choose another container service
Unfortunately we had to install Hashicorp Vault as root user. Fortunately, this does not apply to Authelia. The setup of a rootless docker is now very easy with the command
Create your own user authelia without sudo privileges and log in to your server via SSH with this user. Since you need Systemd for rootless docker, it is not possible to start the docker socket with su - username or sudo -u username.
Authelia can be configured almost entirely via environment variables, with a few exceptions.
The only configuration file needed is for the rules. In part 1 we had already defined a few environment variables in the Hashicorp Vault. Now we want to use this configuration.
First, we log in to the Hashicorp Vault and populate these variables.
Log in to the Vault with your root token and choose the secret engine we have already created (in my case it was Authelia with the path secret). Then click on "Create new version".
All keys ending with key, secret, password, or token correspond to a secret. All others are configuration variables.
Shimmy through key by key. If you get stuck somewhere, you can also use less environment variables and use the instructions here where you will also find a script for generating the passwords for Authelia.
When you're done, click Save and you now have version 2 of your Secrets. Hashicorp Vault always stores a version history of your passwords.
Now create a new folder on your server with a docker-comopose.yml file:
Create Authelia users with hashed passwords as described in the old manual and two folders config and redis.
But since we now configure almost everything via environment variables, we can slim down the configuration.yml considerably.
Starting the container manually with docker compose will not work now and in the future, because your system does not know the environment variables. For this we use Ansible.
Install Semaphore for Ansible
You can also encrypt with the ansible-vault command (please don't confuse it with Hashicorp Vault). Unfortunately, this is not suitable as a secret manager, because in case your password is compromised, an attacker would have all your passwords.
Since we don't want to log into the cli every time to start a container, we install the graphical tool Semaphore for Ansible for this purpose.
The installation of Semaphore is described straightforward in the official manual. Since we want to use Ansible on the guest system, I don't recommend installing the Docker version of Semaphore.
Create a new folder and user for Semaphore. Since we have installed Authelia Docker rootless mode, we don't need root permissions for Ansible, but it is still best practice to give it sudo rights.
usermod -aG sudo <ansibleUser>
Since we do not want the Ansible user to enter a password for executing commands as an Authelia user, we still need to adjust the sudo settings for the Ansible user.
Open the sudo settings with the following command:
<ansibleUser> ALL=(<autheliaUser>) NOPASSWD: ALL
below the line starting with "%sudo".
Create a local Git repository
Semaphore requires a Git repository to manage local playbooks.
As an Ansible user, create a new folder semaphore.git:
The directory tree should now look like this:
Explanation: semaphore.git is the directory for your Git repository. The semaphore directory will contain the playbooks for Ansible.
i always recommend a dependency manager like python poetry, but you can also just install ansible via apt install ansible.
Other Python packages that are required are:
Install them each via python pip or poetry add.
Create an Ansible role:
~/semaphore$ ansible-galaxy init --init-path roles/ start_docker_container
- Role start_docker_container was created successfully
We need two Ansible collections:
- community.docker : for Docker Compose management
- community.hashi_vault : for fetching the secrets from the Hashicorop vault.
ansible-galaxy collection install community.hashi_vault
ansible-galaxy collection install community.docker
Create an Ansible playbook to launch the containers:
Your directory tree should now look like this:
Most of the files were created automatically and we need to customize them.
First open the file main.yml directory ~/semaphore/roles/start_docker_container/tasks
Now adjust the file main.yml directory ~/semaphore/roles/start_docker_container/defaults
Do these environment variables look familiar? They are the same as in the Vault and in the docker-compose file.
Ansible vault encryption
So far, however, the Ansible role does not know where and how to log into the Hashicorp Vault. Of course, we don't just store this login password in the script, but encrypt it with Ansible Vault.
We need three encoded parameters:
Encryption is done in each case using the ansible-vault encrypt_string 'YOURVAULTPASSWORD' --name vaultpassword command . Thus:
With each input you get a new encrypted string, which you have to enter additionally into the defaults/main.yml of the role.
It is important that you always enter the same Vault password, otherwise the encryption result will differ. Of course, keep this password safe, e.g. in Keepass.
Once you have everything done, you still need to write changes to the local Git upstream:
The default port for Semaphore is 3000, log in to the GUI.
Create a new project.
Add your vault key to the key store.
And create a local key without a password since semaphore wants authentication even on localhost.
Add your created git upstream path as a new repository.
And add an empty environment file.
And another empty inventory file since we just want to work on localhost.
Now we go to the actual work. Semaphore works with task templates. The Ansible playbooks must be located in the root directory of the repostory. They will not be found in directories below that. This is in accordance with the Ansible defaults.
Start a task.
If everything goes well you will see a bright green.
How to add more containers
Basically, for each container, you just need to find the environment variables and set them in the Hashicorp Vault. You don't necessarily need to create a new secret engine for each user. The ACLs are fine-grained and very customizable.
You can extend the automation in Ansible with tags, e.g. docker1, docker2 and restrict the playscript with tags only to the container you want to launch.
A few caveats and why it doesn't matter
Every security measure comes at the expense of usability. This is also the case here. If your server reboots, you will have to unseal your vault every time.
The same applies to the Docker containers, if they should crash, they would no longer be able to be restarted without further action.
Automation with Ansible makes things easier. The admittedly somewhat odd approach with the local Git repository at Semaphore complicates this somewhat unnecessarily. If you change something in the playbooks, you have to commit and push the change to the directory first. But the Semaphore interface is just an icing on the cake and you don't have to rely on it.
The security of this system thrives on unsealing and sealing the Hashicorp Vault. If someone were to get your Ansible Vault password and gain your Authelia user's readonly access to the Hashicorp Vault, they would have nothing to gain if the Vault is sealed.
If your server or hard drive is lost, they could still get your passwords and secrets.
Of course, password protection alone is not enough to secure your system, but it is a guaranteed advantage against attackers.
At the end of the day, you always have to work and live in IT in a way that the attacker has already penetrated your system and is looking over your shoulder. In this case, however, it wouldn't do him any good.
Environment variables, as long as you have not stored them in a bashrc or similar, live only as long as a shell is open. In this case, it means that once the Ansible job is finished, the environment variables only live in the container in non-persistent memory. Since containers are inherently immutable, they disappear when the container is restarted.