Photo by Cristina Gottardi on Unsplash

Secure servers with SaltStack and Vault (part 3)

Creating single-use database credentials.

Glynn Forrest
Wednesday, September 19, 2018

In the last 2 stages we used SaltStack to provision a Vault service and create some policies. Now let’s use this foundation to secure a common shared resource - a database.

The power of Vault is in its ability to create temporary, dynamic credentials for shared resources, with the option to have them automatically expire at a certain time, or on-demand with explicit revoking. If a dynamic credential is ever stolen by a malicious party, a Vault operator can revoke access to the database without locking out valid users and applications.

Let’s install a PostgreSQL database using Salt and restrict access to users that are granted the read-only Vault policy we created in part 2. Whenever a user wants to access the database, they will ask Vault for temporary credentials. Once granted, they can log in to the database and perform their work, but only while the credentials haven’t expired. After this, the user will have to ask Vault to gain access again.

Install PostgreSQL

Create salt/postgres/install.sls with the following:

postgres:
  pkg.installed:
    - names:
      - postgresql-9.6
      - postgresql-client-9.6
  postgres_database.present:
    - name: secure-servers
    - require:
      - pkg: postgres
  postgres_user.present:
    - name: vault
    - superuser: True
    - password: complex_p$ssw0rd
    - require:
      - pkg: postgres

This will install PostgreSQL, create a database, and create a user for Vault to login as.

Run the state with salt-call, checking we can connect as the Vault user with psql:

sudo salt-call state.sls postgres.install

psql -h 127.0.0.1 -U vault -W -d secure-servers -c '\list'
                                    List of databases
      Name      |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
----------------+----------+----------+-------------+-------------+-----------------------
 postgres       | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 secure-servers | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0      | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
                |          |          |             |             | postgres=CTc/postgres
 template1      | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
                |          |          |             |             | postgres=CTc/postgres
(4 rows)

After typing in the vault user password (complex_p$ssw0rd in this case), we should be connected to PostgreSQL and see our newly created database, secure-servers.

Note that in the psql command we are connecting over TCP (127.0.0.1), not the local socket (localhost). With the default setup in Debian, database users without a corresponding system account can only connect over local TCP, as shown in /etc/postgresql/9.6/main/pg_hba.conf:

# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5

For a production setup, Vault and PostgreSQL will be on different machines, so you’ll need to tweak pg_hba.conf to allow the user to connect. See the fantastically comprehensive PostgreSQL documentation for more details.

The Vault database secrets engine

The instructions for mounting a PostgreSQL database engine into Vault are quite straightforward, but manual. Since we’re using SaltStack, let’s automate it instead.

Unfortunately, the current Vault state module is quite limited, only supporting policy creation. Let’s write a custom state module that will interact directly with the Vault API.

Enable the database secret backend

At the terminal, step 1 would be to run vault secrets enable database. Behind the scenes the vault client makes an API request to the Vault server, and that’s what we’ll do in our state module.

Create salt/_states/ss_vault.py and add the following:

from __future__ import absolute_import

import logging
import salt.utils

log = logging.getLogger(__name__)

def database_backend_enabled(name):
    """
     Equivalent to 'vault secrets enable database'
    """
    ret = {
        'name': name,
        'changes': {},
        'result': True,
        'comment': '',
    }

    try:
        url = "v1/sys/mounts/{}/tune".format(name)
        response = __utils__['vault.make_request']('GET', url)
        if response.status_code == 200:
            ret['comment'] = "Database backend is already enabled"
            return ret

        url = "v1/sys/mounts/{}".format(name)
        data = {
            "type": "database",
            "description": "Database backend for secure servers"
        }
        response = __utils__['vault.make_request']('POST', url, json=data)
        if response.status_code != 200:
            response.raise_for_status()

        ret['comment'] = "Enabled backend of type 'database' at /{}".format(name)
        ret['changes'] = {
            name: "enabled"
        }
        return ret
    except Exception as err:
        ret['result'] = False
        msg = '{}: {}'.format(type(err).__name__, err)
        log.error(msg)
        ret['comment'] = msg

        return ret

That’s a lot of code to replace a single command, but most of it is SaltStack boilerplate to ensure it is idempotent (can be run many times with the same result every time). First we make a GET request to v1/sys/mounts/<mountpoint>/tune to check if the secret backend has been mounted yet. If not, only then do we make a POST request to v1/sys/mounts/<mountpoint> to enable it. The requests are made with the Vault helpers built into Salt at salt.utils.

Since salt modules don’t have namespaces, we’ve prefixed the custom state file with ss_ (secure servers) to prevent any confusion with the official Vault state modules.

Enable a database connection

Now we want to run the equivalent of:

 vault write database/config/my-database \
    plugin_name="..." \
    connection_url="..." \
    allowed_roles="..." \
    username="..." \
    password="..."

with the salt state.

Add a new method to salt/_states/ss_vault.py called postgres_database_enabled:

def postgres_database_enabled(name, connection_url='', username='', password='', allowed_roles='', mount_point='database'):
    """
     Equivalent to 'vault write database/config/<name>'
    """
    ret = {
        'name': name,
        'changes': {},
        'result': True,
        'comment': '',
    }

    data = {
        "plugin_name": "postgresql-database-plugin",
        "allowed_roles": allowed_roles,
        "connection_url": connection_url,
        "username": username,
        "password": password,
    }

    try:
        path = "/{}/config/{}".format(mount_point, name)
        url = "v1"+path
        response = __utils__['vault.make_request']('GET', url)
        if response.status_code == 200:
            ret['comment'] = "Database is already enabled at {}".format(path)
            return ret

        response = __utils__['vault.make_request']('POST', url, json=data)
        if response.status_code != 200:
            response.raise_for_status()

        ret['comment'] = "Enabled database at {}".format(path)
        ret['changes'] = {
            name: "enabled"
        }

        return ret
    except Exception as err:
        ret['result'] = False
        msg = '{}: {}'.format(type(err).__name__, err)
        log.error(msg)
        ret['comment'] = msg

        return ret

In a similar style to the previous method, we check if the database config exists already with a GET request, enabling it with a POST request if not.

This method is pretty naive - if the database configuration exists but is different from the desired configuration, it will do nothing.

Create database roles

Finally, we want to replicate

vault write database/roles/my-role \
    db_name=my-database \
    creation_statements="..." \
    default_ttl="1h" \
    max_ttl="24h"

in our state.

Add another method to salt/_states/ss_vault.py, postgres_database_enabled:

def postgres_role_enabled(name, db_name='', creation_statements=None, mount_point='database', default_ttl=None, max_ttl=None):
    """
     Equivalent to 'vault write database/roles/<name>'
    """
    ret = {
        'name': name,
        'changes': {},
        'result': True,
        'comment': '',
    }

    try:
        path = "/{}/roles/{}".format(mount_point, name)
        url = "v1"+path
        response = __utils__['vault.make_request']('GET', url)
        if response.status_code == 200:
            ret['comment'] = "Role is already enabled at {}".format(path)
            return ret


        data = {
            "db_name": db_name,
            "creation_statements": "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";",
            "default_ttl": default_ttl,
            "max_ttl": max_ttl,
        }
        response = __utils__['vault.make_request']('POST', url, json=data)
        if response.status_code != 200:
            response.raise_for_status()

        ret['comment'] = "Enabled role at {}".format(path)
        ret['changes'] = {
            name: "enabled"
        }

        return ret
    except Exception as err:
        ret['result'] = False
        msg = '{}: {}'.format(type(err).__name__, err)
        log.error(msg)
        ret['comment'] = msg

        return ret

Here we give Vault the actual SQL statements used to create database users. Thanks to PostgreSQL’s VALID UNTIL syntax, we can specify when the account expires. For other databases, Vault will check-in periodically to remove expired accounts.

Like before, there are a few shortcomings with this state function (not checking for changed configuration, hardcoded SELECT only database grants, hardcoded public schema) that should be fixed before using this state in the real-world.

A state file using the state module

Create salt/postgres/vault.sls with the following:

database_backend:
  ss_vault.database_backend_enabled:
    - name: database

postgres_connection:
  ss_vault.postgres_database_enabled:
    - name: secure-servers
    {% raw %}
    - connection_url: "postgresql://{{username}}:{{password}}@127.0.0.1:5432/secure-servers"
    {% endraw %}
    - allowed_roles: read-only
    - username: vault
    - password: complex_p$ssw0rd

readonly_role:
  ss_vault.postgres_role_enabled:
    - name: guest
    - db_name: secure-servers
    # can login for 5 minutes by default, lease renewable up to 1hr
    - default_ttl: 300
    - max_ttl: 3600

Some things to note:

By setting allowed_roles to read-only, only Vault users with that role have access.
We’re using the {% raw %} and {% endraw %} jinja tag pair to avoid it interpolating the {{ and }} tags in the connection url string.
default_ttl and max_ttl can be an integer amount of seconds or a string such as "24hr" or "1hr", but I found using strings returned a 400 Bad Request error from Vault.

Running the state

Refresh the module cache to tell Salt about our new state module, then run the vault.sls state:

sudo salt-call saltutil.sync_all
sudo salt-call state.sls postgres.vault

Not quite! The Vault token the Salt minion has doesn’t have permission to hit sys/mounts/* or /database/* endpoints, so you’ll see an error.

Update the policy it uses by adding more paths to salt-minions.hcl

path "sys/mounts/*" {
  capabilities = ["read", "list", "create", "update", "delete"]
}

path "database/*" {
  capabilities = ["read", "list", "create", "update", "delete"]
}

and writing it into Vault again:

vault policy write saltstack/minions salt-minions.hcl

As we said in the last post, you may want to automate the provisioning of this policy with a script external to Salt.

Now run the state again:

sudo salt-call state.sls postgres.vault

Success! We’ve told Vault about the PostgreSQL database and created a role.

Getting some credentials

With all that setup, we can now ask Vault for some database credentials.

Authenticate as a read-only Vault user (see part 2):

vault auth enable userpass
vault write auth/userpass/users/test password="test" policies="read-only"
unset VAULT_TOKEN
vault login -method=userpass username=test

and then ask Vault for some database credentials:

vault read database/creds/guest

Key                Value
---                -----
lease_id           database/creds/guest/1e3cec8d-3bd5-5b8d-280c-9648bc6001d8
lease_duration     2m
lease_renewable    true
password           A1a-1YsyqAskdVUjFGBO
username           v-userpass-guest-yB855bVGDgnBEcbI6Yzz-1537295150

Finally, login to the database using psql, remembering that the created users can only access on TCP:

psql -h 127.0.0.1 -U v-userpass-guest-yB855bVGDgnBEcbI6Yzz-1537295150 -W -d secure-servers

Password for user v-userpass-guest-yB855bVGDgnBEcbI6Yzz-1537295150:

psql (9.6.10)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

secure-servers=>

Success! We’ve connected to PostgreSQL using dynamic credentials.

To test they expire, either wait for their lease duration to pass, or revoke them explicitly:

vault lease revoke database/creds/guest/1e3cec8d-3bd5-5b8d-280c-9648bc6001d8

The credentials will no longer work.

A high-level overview

We had to roll our sleeves up and write some python code for this part in the series, so it’s easy to lose sight of the overall goal, which is to provision a database automatically with Salt, avoiding any storage of passwords outside of Vault.

Here is a top down approach of how a SaltStack/Vault/PostgreSQL setup could work:

An operator needs a new PostgreSQL server, so they update top.sls to provision a server with PostgreSQL installed.
During install, Salt uses the default PostgreSQL account to create a new remote database user with a long and complicated random password, then locks down the default account.
Salt makes an API call to Vault, telling it about the new database with vault_ss.postgres_database_enabled, passing it the random password, and creating a role to access it with vault_ss.postgres_role_enabled. Salt will forget about the random password when the state has completed and the process exits.
Vault rotates its database credential using Database Root Credential Rotation, replacing the random password from Salt with a new password that only it knows.
At this point, the only way to administer the database is with root access to the machine or using the remote user account, which only Vault knows the password to.
To get credentials, you authenticate with Vault, which hands you temporary credentials for a use case.

This works well for human users, but not so much for long running applications - we’d have to generate new credentials and redeploy them constantly. There are different tactics to avoid this, such as making your application Vault-aware, or running it on a process scheduler that can interact with Vault and pass it credentials dynamically. We can also use Hashicorp’s Consul for this, which we’ll learn about in part 5.

Conclusion

Great! We’re on the way to running databases with zero credential leakage.

With our custom state module, we’ve enabled the Vault database secrets backend, added a database, and defined a role. It’s a good start, but isn’t really ready for production use, with naive resource checking and many hardcoded options.

Work is ongoing to add more Vault capabilities into Salt, see issue #40043. Perhaps we’ll see that work integrated soon, removing the need for our custom module.

In part 4 we’ll drop our insecure http endpoint and host Vault securely over https.