Announcing knife-briefcase

One of the stumbling blocks for teamwork with Chef is sharing secrets; there are some files and other data that are sensitive, should not be committed to the version control repository, and the admins (or administration scripts) need them in daily work. Such files include secrets for encrypted data bags, private SSL keys, AWS keys (and other access keys and passwords), or private SSH keys to log into newly created EC2 instances. Until now, I kept these in .chef directory, and either had some crude script to pack them in a GPG-encrypted tarball stored somewhere (Dropbox, a file accessible via SSH, whatever), or was just sending around GPG-encrypted keys and files over e-mail.

Then I figured, hey, Chef server already has facilities to store and retrieve small pieces of data. Data bags don’t have to be used by recipes! What if I could just GPG-encrypt the files as I used to, but had a plugin to keep these in the Chef server?

This turns out to work quite nicely. The “knife briefcase” plugin (because you don’t store sensitive paperwork in a bag – you need a proper briefcase!) is available at https://github.com/3ofcoins/knife-briefcase and soon on Rubygems (I want to give it more real world testing first). It uses GPG (via ruby-gpgme) to encrypt and sign the content that is then stored in a data bag – and can be retrieved and decrypted. If list of authorized people changes (as you add new team member, or someobody leaves), knife briefcase reload will re-encrypt all the items to make the change easy. Simple, and effective.

I am aware that there’s already a plugin that serves similar purpose: chef-vault. I even use it in some of my cookbooks. But I don’t like it for this particular use case for two reasons.

First is minor: the command line requires me to always provide a node search query. If I want to share a secret only between admins, I need to provide a fake query that will return no results. It also requires some plumbing to re-encrypt the items. It’s not this much of an issue – it just requires some hacking. If it was the only reason, I’d happily send a pull request rather than write a separate new tool.

The second reason is that it’s complicated to set up a new person. Chef-vault uses administrator’s Chef API secret key to encrypt the data. This means that to re-encrypt the data for new team member, they need to create their API user first, and it requires a back-and-forth:

  1. New user gives me public SSH and GPG keys (which are needed anyway)
  2. I configure SSH access for the provided key and ping them to say they can start
  3. New user is able to create Chef API user now
  4. They ping me, or someone else who has access to the data, to re-encrypt it
  5. They are blocked until this is done

With knife-briefcase, the flow is much more async:

  1. New user gives me SSH and GPG public key (which they need to do anyway)
  2. I configure SSH access for the provided key, use GPG key to re-encrypt the data, and ping them to say they can start
  3. New user is able to create Chef API user and start work right away

This way, there’s one back-and-forth less, and the whole setup feels much simpler to the newbie, who can focus on getting to know the project and getting some work done rather than wait for access to be configured. Complicated setup is my pet peeve, and over last months I did quite a bit of work to simplify the agile administrator’s command center. I think I’m getting close to a nice setup – I should sit down and describe it in a separate blog post someday.

Meanwhile, have fun with knife-briefcase!


Installing Vagrant plugins from local files

tl;dr: to install a Vagrant plugin locally before publishing it to Rubygems, use bundle cache and create a local rubygems index.

I have just bumped into a weird problem: I’m creating an Omnibus package, and I want to run builders for different systems with Vagrant. That’s supported and should run out of the box; but life would be too easy if it worked that way. There were some Gem updates, some dependency mess I’m too lazy to untangle, and as I tried to start vagrant up, I ran into an obscure backtrace (and I wasn’t the only person that saw this error). Seems like a new problem, and there’s already an issue open for it on vagrant-berkshelf project.

Upgrade to Berkshelf 2.0. With a bit of luck, it should be a one-liner. Clone, bundle, run tests; upgrade Berkshelf in .gemspec, bundle, run tests. They pass. Nice. Let’s try with my installed Vagrant… now, how do I install the patched plugin?

Let’s see…

Oh, well. According to Vagrant docs, *the* supported way to install plugins is by publishing them on Rubygems. I don’t want to publish it yet, I’ve just created an untested patch!

The vagrant plugin install command can accept file name. I suppose the file should be a built gem (the documentation doesn’t mention that), so let’s try. Sweet, now it fails to fetch the gem’s dependencies.

There also is a --plugin-source PLUGIN_SOURCE option, documented as Add a RubyGems repository source. If I generate a rubygems index for my single gem, it still fails to find dependencies. Looks like it rather means Use specified RubyGems repository source. I need all dependencies in place, together with my built gem. Bundler had such an option, I think…

Solution!

  • Build a .gem file with your plugin (for vagrant-berkshelf, it was thor gem:build).
  • Run bundle cache to have all dependencies as .gem files in vendor/cache directory.
  • Copy built pkg/vagrant-berkshelf-1.2.0.gem gem to vendor/cache
  • cd vendor/cache
  • Build a RubyGems repository index: gem generate_index
  • Install the plugin out of local RubyGems repository: /usr/bin/vagrant plugin install vagrant-berkshelf --plugin-source file://pwd

I needed to run /usr/bin/vagrant explicitly, because I let Bundler install Vagrant as a local Ruby gem and put it in my path before system-wide Vagrant. If you do bundle install --path=somewhere/else (I usually go with vendor/gems), you won’t need to do this.

I hope this will save someone else an hour or two of their life, or even a gray hair – maybe it will be myself a month from now, trying to remember what was the magic there!