Secure servers with SaltStack and Vault (part 2)

• Part 1
• Part 3
• Part 4
• Part 5
• Supporting files

In the previous post we installed SaltStack and Vault, and wrote some simple secrets using the initial root token.

Using the root token for everything is bad practice, so in this post we’ll use policies and authorization to restrict access to Vault.

As before, the files for this series are available in this github repo.

A read-only user

Let’s start with a simple use-case: a token that grants read-only access to all secrets.

Create a secret with the initial root token:

export VAULT_TOKEN=<root-token>
vault write secret/password value=hunter2

Now create a policy file, read-only.hcl:

path "*" {
  capabilities = ["read", "list"]
}

And add the policy to Vault:

vault policy write read-only read-only.hcl

Then create a new token, granting it the read-only policy.

vault token create -policy=read-only

Key             Value
---             -----
token           ee6cf5a4-05dc-60e7-84e0-c571d60d0dc6
token_accessor  ee0a6b6a-4a44-dd73-64d8-e6e5728bc699
token_duration  768h0m0s
token_renewable true
token_policies  [default read-only]

We can see the new token, duration, and other properties in the output.

Now use the new token to grant read-only access to the console. We can read secrets, but writing to them is disallowed.

export VAULT_TOKEN=ee6cf5a4-05dc-60e7-84e0-c571d60d0dc6
vault read secret/password

Key                     Value
---                     -----
refresh_interval        768h0m0s
value                   hunter2

vault write secret/password value="something"

Error writing data to secret/password: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/secret/password
Code: 403. Errors:

* permission denied

Managing policies and tokens with SaltStack

Of course, creating policies manually isn’t in the spirit of configuration management. Let’s use Salt to create policies for us.

Create the salt state salt/vault/policies.sls:

vault_policy_read_only:
  vault.policy_present:
    - name: read-only
    - rules: |
        path "*" {
          capabilities = ["read", "list"]
        }

Use the rules key to specify the HCL for the policy.

Running this state currently won’t work - we need to configure salt to access vault first.

Granting the salt master access to vault

To interact with vault, the salt master needs a token that allows it to create tokens for minions. Whenever a minion needs to interact with vault, it will use salt’s peer runner system to get a vault token to use.

For example, salt vault-minion state.sls vault.policies will instruct the vault-minion salt minion to run the above salt state. To authenticate to vault, it will ask the salt master for a token.

To be able to create new tokens, the salt-master needs to be granted a suitable policy. Unfortunately this reveals a chicken-and-egg problem: to create tokens with salt, salt needs to have a token with a suitable policy! We will create a policy manually for now, and in future articles learn how to automate it.

Create salt-master.hcl:

path "auth/*" {
  capabilities = ["read", "list", "sudo", "create", "update", "delete"]
}

Adding capabilities for the auth/* path allows the salt-master to create other tokens.

Add the policy, and generate a token for the salt master.

vault policy write salt-master salt-master.hcl
vault token create -policy=salt-master

With the new token created, configure the master to connect to vault. Add the following to /etc/salt/master.d/vault.conf:

vault:
  url: http://127.0.0.1:8200
  auth:
    method: token
    token: <generated_salt_token>

Granting salt minions access to vault

Each minion needs access to the vault runner functions on the master. All runners are disabled by default in salt, so we’ll need to grant them access. Add to /etc/salt/master.d/peer_run.conf:

peer_run:
  .*:
    - vault.generate_token

Then restart the salt-master.

sudo service salt-master restart

This will allow all minions to generate tokens. You may want to restrict it to certain minions; replace .* with another minion target instead (e.g. vault-minion).

Now use a minion to read a secret from vault:

sudo salt-call vault.read_secret 'secret/password'

The generated token doesn’t have access! We need to grant a policy for the minion tokens.

Create salt-minions.hcl and add it as a policy:

path "secret/*" {
  capabilities = ["read", "list"]
}

vault policy write saltstack/minions salt-minions.hcl

Note the policy name: saltstack/minions. Minion tokens have the saltstack/minions and saltstack/minion/{minion} policies by default, but you can customize this in the salt master configuration. See the vault module documentation for more details.

Now the minion can read from vault!

sudo salt-call vault.read_secret 'secret/password'

stretch.localdomain:
    ----------
    value:
        hunter2

Apply policies

Finally, we can now apply our vault.policies state.

sudo salt-call state.sls vault.policies

[ERROR   ] Failed to get policy: 403 Client Error: Forbidden for url: http://127.0.0.1:8200/v1/sys/policy/read-only

Not quite! We need to tweak the saltstack/minions policy to allow the creation of new policies. Update salt-minions.hcl and send it to vault again:

path "secret/*" {
  capabilities = ["read", "list"]
}

path "sys/policy/*" {
  capabilities = ["read", "list", "create", "update", "delete"]
}

vault policy write saltstack/minions salt-minions.hcl

At last, we can define our policies with salt.

sudo salt-call state.sls vault.policies

Authentication backends

So far we’ve used tokens to authenticate to Vault. This works at the console and with Salt, but isn’t practical for other usage.

Vault supports many authentication backends, allowing you to authenticate with tokens, passwords, directory services such as LDAP, AWS IAM roles, etc.

Let’s mount the userpass backend and create a test user with read only access.

vault auth enable userpass
vault write auth/userpass/users/test password="test" policies="read-only"

To test it out, unset the VAULT_TOKEN environment variable and login.

unset VAULT_TOKEN
vault login -method=userpass username=test

Password (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  081bc2ae-9ece-16e5-4588-2dcad5a6caab
token_accessor         a5cb2a81-d5c5-0bac-49b9-36ea9e9fff5c
token_duration         768h
token_renewable        true
token_policies         ["default" "read-only"]
identity_policies      []
policies               ["default" "read-only"]
token_meta_username    test

Success! We’ve authenticated using the userpass auth backend and have the read-only policy.

Conclusion

In this post we created different policies for interacting with vault. We learned how to generate tokens with access to these policies, and how to use vault from salt minions.

In the process of setting up, we had to create the salt-master and saltstack/minions policies manually. A production setup would create these policies automatically when vault is installed. We can’t use salt for this initially, but something like a bash script may be sufficient to bootstrap the policies salt needs.

Check out Hashicorp’s example script here.

In part 3 we’ll look at generating secrets dynamically for databases. See you then!