Secure servers with SaltStack and Vault (part 4)

• Part 1
• Part 2
• Part 3
• Part 5
• Supporting files

So far in this series we’ve installed Vault, created some policies, and used it to provide temporary credentials to access a database. Now let’s use SaltStack’s x509 certificate capabilities to serve Vault over https.

Create a self-signed trusted root certificate

Create salt/ca/init.sls with the following:

ca_x509_module:
  pkg.installed:
    - name: python-m2crypto
    - reload_modules: True

ca_root_key:
  file.directory:
    - name: /etc/pki
    - mode: 0755
  x509.private_key_managed:
    - name: /etc/pki/root.key
    - bits: 4096
    - require:
      - pkg: ca_x509_module
      - file: ca_root_key

ca_root_cert:
  x509.certificate_managed:
    - name: /etc/pki/root.crt
    - signing_private_key: /etc/pki/root.key
    - CN: {{grains['id']}}
    - basicConstraints: "critical CA:true"
    - keyUsage: "critical cRLSign, keyCertSign"
    - subjectKeyIdentifier: hash
    - authorityKeyIdentifier: keyid,issuer:always
    - require:
      - x509: ca_root_key

ca_trust_root_cert:
  file.managed:
    - name: /usr/local/share/ca-certificates/root.crt
    - source: /etc/pki/root.crt
    - makedirs: True
  cmd.run:
    - name: 'update-ca-certificates --fresh'
    - onchanges:
      - file: ca_trust_root_cert

This will use Salt’s x509 state module to create a self-signed root certificate, importing it into the system’s trust store.

Run the state with salt-call, then check the certificate has been created and is trusted by the vagrant VM:

sudo salt-call state.sls ca

openssl verify /etc/pki/root.crt
# /etc/pki/root.crt: OK

This is just a basic use of Salt’s x509 module, as the root certificate is generated and trusted on one machine only. In a real world situation we will have multiple minions needing to trust the root certificate. There are various ways to distribute the certificate, but the Salt Mine is often a good solution. See Using SaltStack for internal SSL certificates for more information.

Updating Vault to use SSL

To host its API over SSL, Vault will need a certificate of its own and a tweak of the configuration file.

Create a new certificate signed by the root certificate at the bottom of salt/vault/install.sls:

vault_ssl_key:
  x509.private_key_managed:
    - name: /etc/pki/vault.key
    - bits: 4096
    - user: vault
    - group: vault
    - mode: 0700

vault_ssl_cert:
  x509.certificate_managed:
    - name: /etc/pki/vault.crt
    - public_key: /etc/pki/vault.key
    - signing_cert: /etc/pki/root.crt
    - signing_private_key: /etc/pki/root.key
    - CN: 'vault.local'
    - subjectAltName: 'IP:127.0.0.1'
    - user: vault
    - group: vault
    - mode: 0400
    - append_certs:
        - /etc/pki/root.crt
    - require:
      - x509: vault_ssl_key

Signing the certificate is straightforward in our case, as the root certificate is on the same machine. Usually Vault will be on a different machine and you’ll have to use the x509 state’s remote signing capabilities to get a certificate. Again, see this previous post for more information.

In the vault_service state id, add the ssl certificate to the list of changes requiring a reload of Vault:

  vault_service:
    ...
    cmd.run:
      - name: 'systemctl reload vault'
      - onchanges:
        - file: vault_config
        - file: vault_service
+       - x509: vault_ssl_cert

Whenever the certificate changes, Vault will reload the certificate without restarting.

Update salt/vault/vault.hcl:

  listener "tcp" {
    address = "127.0.0.1:8200"
-   tls_disable = 1
+   tls_cert_file = "/etc/pki/vault.crt"
+   tls_key_file  = "/etc/pki/vault.key"
  }

This tells vault to listen over https using the generated certificate and key.

Run the vault.install state file again:

salt-call state.sls vault.install

Vault should now be listening over https, its configuration reloaded without having to unseal it again.

Update your Vault client to use https and verify connectivity:

export VAULT_ADDR=https://127.0.0.1:8200
vault status

You could also use curl to check the SSL handshake is working correctly:

curl -v https://127.0.0.1:8200

Vault’s PKI backend

Vault can also be used as a certificate authority thanks to its PKI secrets engine.

With Salt’s x509 state module, it’s possible to create an intermediate certificate, mount the Vault PKI engine, and pass the intermediate certificate into Vault.

With the engine mounted:

vault secrets enable pki

and the intermediate certificate created:

# ...
vault_intermediate_cert:
  x509.certificate_managed:
    - name: /etc/pki/vault-ca.crt
    - signing_policy: (configued_salt_x509_policy)
    - CN: 'Vault Intermediate CA'
# ...

the certificate can be added to Vault with the /config/ca PKI endpoint:

cat /etc/pki/vault-ca.key /etc/pki/vault-ca.crt > /etc/pki/vault-ca.pem
vault write pki/config/ca pem_bundle=@vault-ca.pem

You can then dynamically request certificates signed by the Vault intermediary:

vault write pki/roles/nomad allowed_domains=example.local allow_subdomains=true max_ttl=24h
vault write pki/issue/nomad common_name=app1.example.local

Vault’s PKI backend is a dense topic, see the documentation for more information. It’s possible to automate these commands using Salt, perhaps by adding to the salt/_states/vault_ss.py state module from the previous post in this series.

Salt’s x509 state module vs Vault’s PKI backend

With both SaltStack and Vault acting as certificate authorities, when is it appropriate to use each? There are many ways to do it, but my approach is the following:

• Use Salt to create the top level root certificate authority, making it easy to distribute and trust this certificate on every machine.
• Use Salt to create an intermediate certificate for Vault’s PKI backend. Certificates signed by this intermediary will be trusted by all machines because the chain of trust leads back to the root certificate.
• Use Salt to create ‘low level’ infrastructure certificates for Vault’s API, Consul, etc. Even though it’s possible, having Vault manage the certificate for its own API can lead to confusion.
• Use Vault’s PKI backend for short-lived certificates, such as applications running in Docker containers or client certificates to authenticate to a service.

Salt provides the foundation for Vault, and Vault generates the majority of certificates for applications to use.

Conclusion

Internal certificate infrastructure can seem daunting at first, but combining SaltStack and Vault allows for a powerful and secure system with maximum control.

Stay tuned for part 5, where we’ll switch to using Consul for Vault’s data store and use Consul Template to dynamically update a configuration file.