Building a SaltStack development machine

Like many great infrastructure projects, SaltStack is open source and regularly accepts code contributions from the community.

I think it’s important to give back to the open source tools, so when I discovered various bugs in Salt’s x509 state module I decided to open a pull request to improve its reliability and ease of use.

Unfortunately, Salt has a large and complicated test suite that can be difficult to run locally. The tests require a lot of Python dependencies, and the tests themselves can make changes to the host system.

This is a perfect use case for a disposable virtual machine. In this post we’ll use Hashicorp Vagrant and SaltStack itself create a machine that can run Salt’s test suite easily and safely.

The full source code for this post is available here.

Init

Make sure Vagrant is installed on your machine, see the installation guide.

Let’s start by spinning up a blank virtual machine. We’ll use Ubuntu 18.04, one of the operating systems used in Salt’s Jenkins infrastructure. In a new folder, create Vagrantfile then power up the machine:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"
  config.vm.hostname = 'salt-test-box'

  # Give the machine 2048MB of RAM.
  # This assumes you're using the virtualbox provider, see
  # https://www.vagrantup.com/docs/providers/ for information on the
  # other providers.
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
  end
end

vagrant up

Let’s write some provisioning code while the ubuntu/bionic64 image is downloading and the machine is booting.

Configuring the Salt provisioner

While this machine will be used to work on the Salt codebase, there’s nothing stopping us from provisioning the machine itself with Salt!

Add to Vagrantfile:

  config.vm.synced_folder "salt", "/srv/salt", :nfs => true
  # required for nfs shared folder
  config.vm.network "private_network", type: "dhcp"

  config.vm.provision :salt do |salt|
    salt.install_type = "stable"
    salt.masterless = true
    salt.minion_config = "salt/minion"
    salt.run_highstate = true
  end

Create salt/minion:

file_client: local

Create salt/top.sls:

base:
  '*':
    - provision

And finally, create salt/provision.sls:

test:
  file.managed:
    - name: /home/vagrant/test.txt
    - contents: 'Provisioned with Salt'

This is the bare minimum setup required for provisioning a Vagrant VM with Salt.

Hopefully the machine has downloaded and booted by now. Tell Vagrant to provision the machine:

vagrant reload --provision && vagrant ssh

Check the file we defined in provision.sls has been created:

cat ~/test.txt
# Provisioned with Salt

Provisioning the machine

Now that we’ve tested a simple state, let’s update salt/provision.sls to install everything we need.

This code is a rough application of the steps written in Salt’s testing guide.

Start with installing required packages:

required_packages:
  pkg.installed:
    - names:
      - git
      - python3-pip

Then checkout the codebase:

codebase:
  git.latest:
    - name: https://github.com/saltstack/salt.git
    - target: /home/vagrant/salt
    - user: vagrant

And use pip3 to install the required dependencies:

python_requirements:
  pip.installed:
    - requirements: /home/vagrant/salt/requirements/tests.txt
    - bin_env: /usr/bin/pip3

Note the use of bin_env to use pip3.

Let’s run the state! As well as using vagrant provision or vagrant reload --provision, we can simply run salt-call without reloading the machine. We’ll include some logging to see what’s going on, especially because some of these operations will take a while.

sudo salt-call state.apply --log-level info

Running tests

Use python3 to execute the runtests.py script. Make sure to run python3 tests/runtests.py --help before anything else to see the full list of options. Running the script without arguments will start the entire testsuite, which can take a while!

Start by running a single unit test:

python3 tests/runtests.py --name unit.modules.test_ps

Then move onto the full collection of unit tests.

python3 tests/runtests.py --unit

Here’s the test I’m interested in, the x509 integration test:

python3 tests/runtests.py --name=integration.states.test_x509

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Starting integration.states.test_x509 Tests
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# sss
# ----------------------------------------------
# Ran 3 tests in 0.023s

# OK (skipped=3)

# -> integration.states.test_x509.x509Test.test_cert_signing  ->  Skip when no M2Crypto found
# -> integration.states.test_x509.x509Test.test_issue_49008   ->  Skip when no M2Crypto found
# -> integration.states.test_x509.x509Test.test_issue_49027   ->  Skip when no M2Crypto found

Hmmm, the tests were skipped due to a missing dependency. You’ll often find this since Salt has functionality for such a wide range of tools. We need to install the m2crypto Python package.

Unfortunately M2Crypto isn’t available for Python 3 yet, so we’ll have to run the tests using Python 2.

Let’s update the list of installed packages and add another pip.installed state call to install the requirements for Python 2.

  required_packages:
    pkg.installed:
      - names:
        - git
        - python3-pip
+       - python-pip
+       - python-m2crypto

python2_requirements:
  pip.installed:
    - requirements: /home/vagrant/salt/requirements/tests.txt

Running again with Python 2:

python tests/runtests.py --name=integration.states.test_x509

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Starting integration.states.test_x509 Tests
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# ----------------------------------------------
# Ran 3 tests in 47.598s

# OK

Success!

If you’re feeling brave, you might want to run the entire suite:

python tests/runtests.py

Conclusion

Working with Salt’s testsuite may seem intimidating, but by leveraging Salt to spin up a temporary test environment we can get going quickly. SaltStack use Salt to provision their test setup in the cloud, see the salt-jenkins project.

To be truly productive with a setup like this, you could use another shared folder for /home/vagrant/salt, making it possible to edit the Salt source code on the host machine.