Secure servers with SaltStack and Vault (part 1)

• Part 2
• Part 3
• Part 4
• Part 5
• Supporting files

As Backbeat provisions new servers for customers and upcoming products, it’s a good time to revise some best practices.

No matter your deployment model, be it virtual machines or container orchestrators, public or private clouds, bare metal or serverless, your infrastructure still needs a place to store secrets. These secrets are critical to the daily operation of server infrastructures, including things like database passwords, api keys, and certificates.

Unfortunately it’s all too common to store these secrets in poor places such as git repositories, excel spreadsheets, and developers’ laptops. We have no control over who can see what, and no way of revoking access from rogue actors.

However in 2015 Hashicorp announced Vault, an open source tool which perfectly solves this problem. In this series of posts we’ll use Backbeat’s preferred configuration management tool, SaltStack, to automate the provisioning of a Vault-backed secure server infrastructure. We’ll use another Hashicorp tool, Vagrant, to spin up virtual machines to work with.

This first post will walk through installing a Salt master and minion, and using them to install and initialize Vault.

The files for this series are available in this github repo, but you’ll still need to run the bash commands yourself if you’re following along.

Create a VM and install Salt

Create a new Vagrantfile:

mkdir secure-servers
cd secure-servers/
vagrant init debian/stretch64

Create a folder for Salt states, and tell vagrant to share it with the virtual machine:

mkdir salt

# Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "debian/stretch64"
  config.vm.synced_folder "./salt", "/srv/salt"
end

Start the VM and login.

vagrant up && vagrant ssh

Now we’ll install a Salt master and minion using the salt-bootstrap script. Remember to use the -M option to install both the master and minion. There are more secure ways to setup Salt, but this works for the sake of this tutorial.

sudo apt-get install curl
curl -L -o bootstrap.sh https://bootstrap.saltstack.com
chmod +x ./bootstrap.sh
sudo ./bootstrap.sh -M

Remember: running unknown scripts from the internet as root is inherently dangerous.

The salt-minion will look for a server with the hostname salt to connect to by default. For the sake of convenience, add an entry to /etc/hosts:

127.0.0.1    localhost
127.0.0.1    salt

Now running salt-key will show the minion trying to connect. Run sudo salt-key -A to accept it.

Install Vault

Create salt/vault/install.sls (on the VM or your host, thanks to the vagrant shared folder), and add the following:

{% set vault_version = '0.10.4' %}

vault:
  archive.extracted:
    - name: /usr/local/bin/
    - source: https://releases.hashicorp.com/vault/{{vault_version}}/vault_{{vault_version}}_linux_amd64.zip
    - source_hash: https://releases.hashicorp.com/vault/{{vault_version}}/vault_{{vault_version}}_SHA256SUMS
    - archive_format: zip
    - if_missing: /usr/local/bin/vault
    - source_hash_update: True
    - enforce_toplevel: False
  file.managed:
    - name: /usr/local/bin/vault
    - mode: '0755'
    - require:
      - archive: vault

This will download, extract, and place Vault in /usr/local/bin. We’ve hardcoded the version and architecture here, but a more advanced state could use the map.jinja pattern to account for multiple operating systems.

Now run the state and check the Vault binary is installed:

sudo salt \* state.sls vault.install
vault

Configure Vault

Vault has a -dev server option that automatically unseals the vault and configures an in-memory backend. We’ll skip that however by writing a config file and learning how to unseal it.

Create salt/vault/vault.hcl and add the following:

storage "file" {
  path = "/var/vault/data"
}

listener "tcp" {
 address = "127.0.0.1:8200"
 tls_disable = 1
}

This simple configuration tells Vault to store secrets encrypted on the filesystem and listen on localhost with TLS disabled. The hcl extension stands for Hashicorp Configuration Language, and is a superset of JSON.

Then update salt/vault/install.sls to manage this file:

vault_config:
  file.managed:
    - name: /etc/vault.hcl
    - mode: '0755'
    - source: salt://vault/vault.hcl

Since this is a debian stretch system, we’ll create a systemd unit file to start the service as a system daemon.

Create salt/vault/vault.service:

[Unit]
Description=Hashicorp Vault server
Requires=network-online.target
After=network-online.target

[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/vault server -config=/etc/vault.hcl
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/local/bin/vault step-down
User=vault
Group=vault
TimeoutStartSec=1

[Install]
WantedBy=multi-user.target

And manage it in the Salt state file, setting up other requirements:

vault_user:
  group.present:
    - name: vault
  user.present:
    - name: vault
    - fullname: Hashicorp Vault Server
    - shell: /usr/sbin/nologin
    - groups:
        - vault

vault_data_dir:
  file.directory:
    - name: /var/vault
    - user: vault
    - group: vault
    - mode: '0700'

vault_mlock_grant:
  pkg.installed:
    - name: libcap2-bin
  cmd.run:
    - name: 'setcap cap_ipc_lock=+ep /usr/local/bin/vault'
    - unless: 'systemctl status vault > /dev/null'
    - require:
        - pkg: vault_mlock_grant

vault_service:
  file.managed:
    - name: /etc/systemd/system/vault.service
    - mode: '0700'
    - source: salt://vault/vault.service
    - require:
        - user: vault_user
        - file: vault_config
  service.running:
    - name: vault
  cmd.run:
    - name: 'systemctl reload vault'
    - onchanges:
      - file: vault_config
      - file: vault_service

As well as creating the systemd unit file, there are some other things going on:

• A vault user and group is created for the server to run as.
• The /var/vault directory is created for Vault to use.
• Capabilities are granted to the vault binary to use the mlock syscall as a non-root user, to prevent Vault’s memory from swapping to disk and being accessed by a malicious party.

Note the onchanges entries in the vault_service state id. It tells the vault service to gracefully reload whenever the configuration or service file changes.

Now run the Salt state again:

sudo salt \* state.sls vault.install
vault

And check we can connect to the vault server:

export VAULT_ADDR=http://127.0.0.1:8200
vault status

You should see Vault respond with a 400 error, complaining that the server is not yet initialized. We need to initialize and unseal Vault to start using it.

Unsealing Vault

What makes Vault so powerful as a secret store is the concept of ‘sealing’ it.

Vault’s data is encrypted with a master key, which is constructed from a series of ‘unseal’ keys. When Vault is first started or restarted, it is ‘sealed’ and secrets cannot be read. The only way to read the data is to reconstruct the master key using a process called Shamir’s Secret Sharing and unlock the vault.

This process allows us to have a configurable amount of keys, of which a configurable amount are required to derive the master key.

For example, we may have

• 6 keys in total, 4 of which are required to derive the master key,
• just 2 keys, both of which are required to derive the master key.

For our purposes, we’ll create 3 keys, and require 2 to unseal. Even as an individual, it’s good practice to require multiple keys - an attacker won’t be able to unseal the vault if they manage to steal one of the keys.

However, make sure you don’t lose the keys! The Vault will remain permanently sealed without at least 2 unseal keys to unseal it.

Run vault operator init -key-shares=3 -key-threshold=2 to create a brand new Vault.

vault operator init -key-shares=3 -key-threshold=2

Unseal Key 1: foT3Qz7GwKppgaPr9tFH1UA9xSCvL28zUpyzgMCJiuCc
Unseal Key 2: JxwRoT9WXzfio56PCtoz0WHLLmWxln656C9B59b/E+MP
Unseal Key 3: m+9JVlLgtAYsdpwxRygHfgMZ8bkLB5kaziE87ikRjVQO
Initial Root Token: c357641e-56c3-11d7-6243-2ed161ccc112

Vault initialized with 3 keys and a key threshold of 2. Please
securely distribute the above keys. When the vault is re-sealed,
restarted, or stopped, you must provide at least 2 of these keys
to unseal it again.

Vault does not store the master key. Without at least 2 keys,
your vault will remain permanently sealed.

Read more about seal/unseal keys here.

Now use 2 of the keys to unseal the vault. Run vault operator unseal twice, pasting in an unseal key when prompted. The vault should now be unsealed. Run vault status to check.

Unfortunately, this key sharing system doesn’t work well with automated provisioning - this is by design. Be aware that whenever Vault restarts you’ll need to unseal it using the unseal keys. This is not the case when it reloads however, hence the use of systemctl reload vault in the state file.

Trying it out

We can now read and write secrets in Vault.

Use the Initial Root Token to configure the VAULT_TOKEN environment variable, then write a secret!

export VAULT_TOKEN=c357641e-56c3-11d7-6243-2ed161ccc112
vault write secret/password value=hunter2

Use the read command to read it out:

vault read secret/password

Key                     Value
---                     -----
refresh_interval        768h0m0s
value                   hunter2

You’ll notice Vault has added a refresh_interval, which we’ll learn about in a future post.

Or in json, if you prefer:

vault read -format=json secret/password

{
    "request_id": "adcfebad-8319-e1ff-cd3f-89d772940ec2",
    "lease_id": "",
    "lease_duration": 2764800,
    "renewable": false,
    "data": {
        "value": "hunter2"
    },
    "warnings": null
}

Vault uses Policies to restrict access to secrets depending on the Entity you authenticate as. This also allows for other nice things like auditing, and revoking a particular entity’s access or secrets.

We’ve been using the root token for authentication, which is highly discouraged. Good Vault usage operates on the principle of the minimum access level required, but the root token grants access to everything!

We’ll replace the root token usage with some core policies in the next post.

Next steps

We’ve accomplished step 1 of using Vault in a secure infrastructure. In future posts we’ll cover:

• Authorization and policies
• Dynamic database secrets
• Enabling TLS for the Vault API
• Using Hashicorp’s Consul for high-availability storage, and Consul-template for dynamically configuring applications with secrets stored in Vault.

See you next time in part 2!