Of Containers, Dockers, Rockets, and Daemons

I have started using Docker soon after it showed up. I’ve been running dockerized services in production since well before 1.0. I am still using it quite extensively and recommend it to people.

I have also recently decided to run my next project on FreeBSD. I’ve been playing with this system for quite a while. It purportedly performs better than Linux — definitely feels faster and more reliable. It is engineered, not evolved, and base system is a consistent composition of kernel and user space. It’s been around for 21 years, more than any Linux distribution except Slackware. On top of that it has native ZFS and pf firewall. It’s quite refreshing to work with a system that… just works. Without fuss. Consistently. A system which is not a composition of tens and hundreds independent pieces of software, glued together with duct tape.

There’s just one “but”: Docker runs only on Linux, and I got used to containerized services. Jails are kind of similar, but the toolkits around them are like Linux’ LXC: you wrap entire system in a jail rather than a single service. I spent much time in the last weeks on a research/exploration effort: how much work would it take to run Docker on FreeBSD with jails for isolation, pf for networking, and zfs for storage? (There’s apparently some effort underway, but it seems stalled, and porting and maintaining the whole thing seems like a lot of effort). What would it take to implement some scripting that would quack in a Docker-like enough way to be useful? (Not much less, but I’d have freedom to be more opinionated about the mechanisms; on the other hand, list of features that would make it fully useful seems to converge to Docker’s documentation). All in all, to get things running in a reasonable time frame, I was going to suck it up, settle on nested jails with some ZFS-based tooling (of my own, no existing jail toolkit seems to fully utilize ZFS), and work on actual features. I already started to write some exploratory code.

At this very moment, CoreOS (any association with Oreo cookies purely accidental) announced that they are releasing their own container runtime, named Rocket. The announcement has since been edited (first version was much more aggressive). Docker responded (the text currently online is also edited, and much less defensive than the original). A quick flame war ensued, then passed, there was some name calling and mud slinging, business as usual on the Internet.

While I wouldn’t call Docker fundamentally flawed, the announcement, and my later exploration of Rocket, has shed some light on problems with Docker that I was vaguely aware of, but couldn’t really put my finger on:

  • Docker is implementation-defined. All specs seem to be an afterthought, and it is hard to predict which pieces of specification are stable. For example, Docker’s HTTP API is kind of documented (but not recommended: officially supported “API” is the command-line docker tool I recall having read the previous part, but cannot find it; the sentiments behind it are mirrored in #7538, or by the fact that there still doesn’t seem to be obvious way to express a docker run invocation through the API), but it leaks implementation details all over the place.
  • Docker is a monolyth. There is one binary that is both server and client, which is responsible for building the images, retrieving them from registry, managing volumes, layering filesystems, isolating processes in containers, managing containers as services (which makes it next to impossible to tie Dockerized services into your service manager without using a separate, idle process whose only role is to stay connected to Docker daemon and terminate as soon as the container exits; hello, unnecessary overhead!), configuring NAT, proxying network traffic, and probably a couple more things I didn’t think of.
  • Docker is opaque. I already wrote about HTTP API leaking implementation details, and being actively discouraged. The registry API is even worse. I have spent half an evening trying to figure out how to run a local, authenticated registry (and implement it if necessary). I gave up. Some of the API was documented; many details were buried in behaviour of the docker binary monolyth. It felt like Perl: kind of standarized and documented, but only perl can parse Perl. Seriously, what the hell? Why on Earth can’t downloading tar files work over plain HTTP(S)?

All of that gives a feeling of vendor lock-in and being on a whim of a company, which may or may not be benevolent, who has just received big investment and started to chase enterprise customers and partnerships. And current direction of Docker seems to point towards more complexity, not less. This direction makes all kind of sense for Docker, but it’s not one I really feel comfortable with — especially that until now application container tools were a Docker monoculture.

Enter Rocket. It is developed specification-first, and the implementation is officially a prototype. Specification focuses on one area only, and strictly distinguishes area of responsibility: building an image is separate from image discovery is separate from the image format is separate from container construction is separate from runtime environment. Each piece is independent, and can be used separately. Specification is clear and precise.

I wrote above about spending half an evening trying to figure out just the Docker registry api (together with relationship of registry vs index vs daemon vs client), and giving up. In a similar half-evening I went through Rocket’s app container specification (including image discovery, equivalent of Docker’s registry), was able to ask one of the developers a couple clarifying questions on IRC (CoreOS people were very friendly and helpful), and now I can say I understand how it works well enough to be able to implement it (given enough time, obviously), or reuse pieces in my own projects.

I don’t feel up to porting whole of Rocket to FreeBSD’s ports right now (but who knows? The codebase looks simple and straightforward), but as I am trying to quickly whip up something container-like on top of jails for my own project, I have some already written, well-designed specification I can work with, and some code that I can at least partially reuse. Docker is useless if I don’t port all of it; Rocket is useful even if I don’t use any of their code, and their code is modular enough for me to use only the pieces I want to. And while I am a bit sad that the whole thing began with name calling and mud slinging, and disappointed by Docker’s initial response, I am really happy to see some competition in the container management tools. Even if Docker stays dominant and Rocket doesn’t take off all the way, diversity of tools and preventing monoculture is valuable on its own. I have to admit: I’m pretty excited now.

Where does the distribution end?

Yesterday I’ve had an inspiring Twitter conversation with Miah Johnson. The conversation was long, branchy, and restricted by the 140 character limit. It kept me thinking. It seems the main difference we had was about where does the distribution end, and the userspace begin.

It’s reasonable to expect that if a distribution has a mechanism for preconfiguring packages, automated installation and configuration, and (kind of) configuration management, then one can use it as an end tool to configure the system. Or at least one of the tools in pipeline. Why reinvent the wheel?

For both me and Miah, the experience of trying to get things done with the Debian/Ubuntu toolchain turned out to be an uphill battle. Up a steep hill made of yak hair and duct tape, to be precise. Our conclusions were different, though: Miah wants to use the distribution’s toolchain, or switch to a distribution that has usable tools. This is how stuff should work, after all. I respect and admire that, because myself… I just gave up.

I find the clunky duct tape automation and idiosyncratic distro’s solutions workable, but by that I only mean that 98% of the time I can just ignore it, and the remaining 2% needs just a small nudge to disable a piece of setup or tell the system that I really, really want to do stuff myself, yes, thank you, I know what I’m doing.

Case in point: debconf-set-selections, which started the whole conversation. Only time I needed to use these was when I used Ubuntu’s MySQL package, to set the initial root password. Nowadays I prefer to use Percona Server, which doesn’t set initial password, so I can make Chef set it right after package installation. Otherwise, the only nudge is to disable automatic start of services when package is installed, to let Chef configure it and start it when it’s ready.

Case in point: Python and Ruby libraries. In my view, the distribution’s packages of Python packages and Ruby gems are not meant to be used in user’s applications – they are only meant to exist as dependencies for packaged application written in Python or Ruby. For applications, I just use the base language from a package (and with Ruby I prefer to go with Brightbox patched version), and use Bundler or Virtualenv to install libraries needed by my application.

Case in point: init system. Until systemd arrives, if I need to manage a service that is not already packaged (such as my application’s processes), I don’t even try to write init scripts or upstart configuration. I just install Supervisor or Runit and work from there. Systemd may change that, though, and I can’t wait until it’s supported in a stable distro.

And so on. Distribution’s mechanisms are there, but the way I see it, they are there for internal usage within distribution packages, not for poking and configuring it from the outside. I can enjoy a wide range of already built software that more or less fits together, security patches, wide userbase (which means that base system is well tested and much of the time if I have problems, the solution is a search box away). If I need, I can package my own stuff with FPM, ignoring this month’s preferred toolkit for Debian packagers. Since recently, I can keep my sanity points when I internally publish custom packages and pull other packages from a patchwork of PPAs and projects’ official repositories by using Aptly. I can run multiple instances and versions of a service contained by Docker. And I can happily ignore most of the automation that the distribution purportedly provides, because I simply gave up on it — Chef gets that job done.

Backups suck — a rant

I’m not even mad, I’m just disappointed. I’m tired. Tired of trying to force my cloud-shaped peg through a tape-shaped hole, of custom data formats and convoluted protocols, and of half-assed systems that work well as long as I have just one machine to manage, with just one kind of data on it — unless I want to hack all the management for different data myself. The current state of open source backup software is sad. I have even tried looking at commercial solutions, but couldn’t extract any real information on what’s inside the box from the enterprise marketing copy. Are my expectations unrealistic?

I’m writing this post freshly after a single restore of 370 gigabytes of database that took almost a week of wrestling with broken storage, incomplete archives, interrupted transfers, stuck communication, and — most of all — Waiting for Stuff to Complete, for hours and hours. In fact, much of this post has been written during the Waiting. Many of the issues I have wrestled with were caused by mistakes on my side: misconfiguration, insufficient monitoring that should have detected issues earlier, and the fact that over last month I was not able to pay enough attention to the day-to-day maintenance, which allowed the suckage to accumulate. At the same time, the software systems should automate out the common parts, make it easy to get things right, and be easy debug when they aren’t. All of the backup systems I’ve ever used or seen fails miserably at two or more out of these three points.

But let’s start from the beginning.

Why do we even need backup?

When we hear the word “backup”, we usually imagine a disaster: say, a database server has failed, everything on it has been lost, and we need to get a new one up, as quickly as possible, losing as little data as possible. But this is just one of many cases when backups are helpful.

Actually, for this particular case, backups aren’t even the best tool; online live replication will be quicker to replace the failed piece (just promote the slave to master, and we’re done), and will lose less data (only the replication lag, usually in range of single seconds). The replication slave can be even used to get some load off the main server by responding to read queries that don’t need perfectly synchronized results, such as analytics and reporting.

This often leads to the conclusion that a backup system isn’t needed after all. Come on, it’s 21st century, we have online replication with heartbeat checks to automatically promote the slave. Why would we need a bunch of static archives that take up space to store and time to recover, what is it, 1980s?

But not only disasters are dangerous to data, and keeping an archive of history snapshots can be useful in many other cases, including recovery from PEBKAC problems where data was damaged by someone, or when an application bug sent a DELETE query to the database that got happily replicated to the slave, in realtime. If you have actual backups — a history of static snapshots going back into the past — you won’t be bothered by hearing any of these:

  • Hey, man, that customer just wrote us they can’t see the old comments on their widget listing page. They swear they didn’t click anything (yeah, right), and that the comments were there 17 days ago. Can you bring them back?
  • Our WordPress has just been hacked, and some code has been injected into the PHP files. Can you check when did it happen and give me the latest clean version to compare?
  • I have worked on that Accounting Spreadsheet three months ago, and I must have deleted it when cleaning my desktop. HALP?
  • Can we test the new release on our full production dataset? The database migration transforms every single document, and we hope we got all destructive corner cases, but you know how creative our users are…
  • I’m preparing a report for the investors, and need some growth figures – do you have some records on amount of customer data we keep for the last three years?

Once you have backup policy right, it’s a time machine where nothing of value is truly lost. It is a safety net for the business. Why then the state of backup software now looks even worse than monitoring software in 2011?

21st Century Backup Checklist

What would I expect of a perfect backup system? Besides seeing into the future and storing only the data that I will actually need, instantly available at the moment it’s needed, and compressed not to take any storage space, that is? A decent backup system would be, in no particular order:

  • Using standard formats and tools. I want to have clear and simple recovery procedure if all I have is backup volumes and a rescue boot / recovery CD. Needing an index with encryption keys and content details is still fine, if the system will generate that for me in a readable format.
  • Encrypted and compressed. You can’t trust your datacenter anymore when you’re in the cloud, spread across five data centers, three continents, three hosting providers, and two storage providers. It’s not your company skyscraper’s enforced concrete cellar anymore. I want my backups to be transparent to me, and opaque to the storage provider.
  • Centrally managed, but without bottlenecks. There should be a single place where I can check status of backups across whole system, drill down through a job’s history, trigger a restore or verification, and so on. On the other hand, some of the heavy lifting should be on the node’s side; in particular, if I’m using cloud storage, the node should directly upload the encrypted data to the storage, rather than push the same data to the central location, which would then bounce it to the cloud.
  • Application-specific. I want to be able to use my tools, that are standard enough for me. If I’m backing up MySQL, I prefer xtrabackup to tar.
  • Zero–copy. I don’t want to have to copy the original data to another directory, then tar it up to an on-disk archive, then encrypt the tarball — still on disk — and only then copy it to the storage. This can and should be done online in a pipeline. We work with terabytes of data nowadays, needing double or triple local storage just to do backup is silly.
  • Able to use different storages. I want to be able to use three cheap unreliable storage providers. I don’t want to be locked in any single particular silo. In particular, I don’t want to pretend that my backup storage is made of pools of magnetic tapes and keep a set of intricate scripts pretending that Amazon S3/Glacier, my local disk directory, or a git-annex repository is a tape autochanger.
  • Supporting data rotation. I want to be able to delete old volumes, not just to add data and keep forever. I want to be able to easily specify how I want the rotation to work, and to change my mind later on.
  • Supporting data reshuffling. It should be possible to move a volume between storages: keep fresh data in the same datacenter, at the same time archive it in the cloud, and put monthly snapshots in deep freeze. If I feel like switching storage providers, adjust my expiration scheme (and apply it to already created volumes), or just want to copy data around manually, I should be able to get it done.
  • Secure. No node in the system should access other node’s backed up data. In fact, no node should even access its own historical data without explicit permission. The less node itself can see or has to say, the better. It is especially bad if node has direct access to underlying storage: in case of a break-in, the attacker can not only take down the server, but also delete all its backups (sometimes even other nodes’ backups)
  • Flexible. I want to be able to restore one machine’s backup to another machine. I want to be able to restore just some files. I want to use backup system to keep staging database up to date with production data, or to provision load test infrastructure that mirrors recent production.
  • Scriptable. I want to be able to give my client a “type this one command” or “push that one button” restore instruction, not three paragraphs. This includes restoring production backup to staging database and deleting sensitive data from it.
  • Testable. I want simple way to specify how to verify that I not only have the backups, but I’ll also be able to restore them. In perfect situation, a one-button fire drill that brings back a copy of production environments and checks that it’s readable. And this button is pushed on a regular basis by the monitoring.

Where do we stand now?

Some of currently available systems meet some of the above points.

Bacula is centrally-managed, secure, flexible, can compress volumes, and with some gymnastics it can rotate and reshuffle volumes, be application-specific and do zero-copy backups and restores (though I haven’t managed to do that, or to even see any tutorial or report from anybody who have done that). It fails miserably when it comes to transparency, standard formats, encryption, and storage backends. And I have to pretend to it that all storage I have is on tapes. It’s awfully clunky and hard to debug when something doesn’t work like it’s supposed to. It’s also underdocumented.

Duplicity handles one machine and one dataset, but it is good with encryption, standard data formats, different storage backends, and rotation. I think it can also be made application-specific, though zero-copy backups may not be possible. Without central management, security and flexibility are irrelevant.

Some people have success with BackupPC, but it’s file-centric, too much decision is left to the node being backed up, and it seems to be focused on backing up workstations.

Other systems that I have looked at either are individual low-level pieces of puzzle, are focused on individual machines rather than whole systems, or are overcomplicated behemoths from the 90’s that expect me to think in terms of tape archives. I couldn’t get through marketing copy of commercial solutions, but I don’t have high hopes in these.

There is one more project I forgot to mention when writing this rant yesterday: Obnam. It is built on some very good ideas (deduplication, data always visible as full snapshots even when incremental, encryption). However, it is still focused on a single node, on backing up files (haven’t found any info about application-specific formats and tools), and uses custom, opaque storage format (which seems to be the price for deduplication — a necessary design trade-off). Without any special means, the node can access — and overwrite or delete — its backup history. If we choose to share a repository between many nodes, each node has also access to all other nodes’ backups.

Of all these, Bacula is closest to what I imagine to be a good solution: it got the architecture right (director for job/volume catalog, scheduling, overall control; storage to store files; client to receive commands from director and push data directly to/from storage). On the other hand, its implementation is just unusable. Its communication protocols are custom, opaque, and tied to the particular version; storage layer could have alternative implementations, but it’s practically impossible, as there is no complete specification of the protocol, nor any hints which parts of it are stable, and which can change. The storage, even on disk, is designed in terms of tape archives. There are even no hooks for disk volumes to allow moving the files around in a smarter way (e.g. with git-annex). Its configuration is more fragile and idiosyncratic than Nagios’. The whole system is opaque and debugging it is a nightmare. It’s not scriptable at all (to make a script that just starts a predefined restore job, I had to use Expect on bacula’s CLI console). It uses custom storage formats that cannot be easily restored without setting up whole machinery. It supposedly can do zero-copy application-specific backups, but it requires quite fragile configuration, and I haven’t ever seen a single working sample of that, or report of anybody having done that. All of these issues are buried deep inside Bacula’s implementation and design. It’s hopeless.

I have also learned that there is a fork of Bacula, named Bareos. Not sure whether it’s actually going towards something more transparent, or just maintaining the hopeless design.

What now, then?

Now, I’m trying to estimate how hard would it be to create a proof-of-concept system based on Bacula’s architecture, but using open protocols, standard tools, and encryption: https for communication, client SSL certificates for authentication, tar as a storage format, gnupg for encryption. For an initial implementation, storage could be handed off to S3 (with its automatic Glacier expiration), but using a façade API that would allow to use different backends. In the last few days, I’ve been playing around with Go to try and implement a proof of concept sketch of such system. This direction looks promising.

If you have any remarks, want to add anything, maybe even offer help with development or design of such a system — or tell me that such system already exists, which would be a great news — feel free to use the comment form below, Twitter, or Hacker News. And if you’re by any chance in London at Velocity EU, just catch me around, or look for a BoF session.


Added additional remark about security, mentioned Obnam and Bareos

Distributing confidential Docker images

Here’s my another pet peeve with Docker: the infrastructure for distributing images is simply Not There Yet. What do we have now? There’s a public image index (I still don’t fully get the distinction of index vs registry, but it looks like a way for DotCloud to have some centralized service that’s needed also for private images). I can run my own registry, either keeping the access completely open (with access limited only by IP or network interface), or delegating the authentication to DotCloud’s central index. Even if I choose to authenticate against the index, there doesn’t seem to be any way to actually limit access to the registry — it looks like anyone who has an index account and HTTP(S) access to the registry can download or push images.

It doesn’t seem there is any way in the protocol to authenticate users against anything that’s not the central index – not even plain http auth. Just to get https access, I need to put Apache or nginx in front of the registry. And did I mention that there is no way to move full images between Docker hosts without a registry, not even a tarball export?

I fully understand that Docker is still in development, and that these problems mean that there is not much of bigger showstopper issues, which is actually good. However, this seems to seriously limit usefulness of Docker in production environments; I need to either stop controlling who’s able to download my images, or I need to build image locally on each Docker host — which prevents me from building an image once, testing it, and then using the very same image everywhere.

And the problem with distribution is not only with distributing in-house confidential software. A lot of open source projects run on Java (off the top of my head: Jenkins, RunDeck, Logstash + Elasticsearch, almost anything from Apache Software Foundation…). While I support OpenJDK with all my heart, Oracle’s JVM still wins in terms of performance and reliability; and it’s not allowed to distribute Oracle’s JVM except internally within an organization. I may also want to keep my Docker images partially configured – software is open, but I’d prefer not to publish internal passwords, access keys, or IP numbers.

I hope that in the long run, it will be possible to exchange images in different ways (plain old rsync, distribution via bittorrent, git-annex network, shared filesystems… I could go on and on). Right now, I found only one way, and it doesn’t seem obvious, so I want to share it. Here it is:

Docker’s registry server doesn’t keep any local data; all it knows is in its storage backend (an on-disk directory, or an Amazon S3 bucket). This means it’s possible to run the registry locally (on, and move access control to the storage backend; you don’t control Docker’s access to the registry, but registry’s access to the storage. It may be implemented either as a shared filesystem (GlusterFS, or even NFS), it may be an automatically synced directory on disk, or – which is what I prefer – a shared S3 bucket. Each Docker host runs its own registry, attached to the same bucket, with a read-only key pair (to make sure it won’t be able to overwrite tags or push images). The central server that is allowed to build and tag images is the only one that has write access. Images stay confidential, and there even is a crude access control (read-only vs write access). It’s not the best performance you can get to distribute the images, but it gets the job done until there’s a more direct way to export/import a whole image.

I hope this approach is useful; have fun with it!

Flat Docker images

Docker seems to be the New Hot Thing these days. It is an application container engine – it lets you pack any Linux software into a self-contained, isolated container image that can be easily distributed and run on different host machines. An image is somewhere in between a well built Omnibus package, and a full-on virtual machine: it’s an LXC container filesystem plus a bit of configuration on top of it (environment variables, default command to run, UID to run it as, TCP/UDP ports to expose, etc). Once you have the image built or downloaded, you can use it to start one or many containers – live environments that actually run the software.

To conserve disk space (and RAM cache), Docker uses AUFS to overlay filesystems. When you start a container from an image, Docker doesn’t copy all the image’s files to the container’s root. It overlays a new read/write directory on top of read-only directory with the image’s filesystem. Any writes the container makes go to its read/write image; all the reads of the unchanged files are actually performed from image’s read-only root. The image’s filesystem root can be shared between all its running containers, and a started container uses only as much space as it has actually written. This conserves not only disk space – when you have multiple containers started from one image, the operating system can use the same memory cache for all of them for the files they share. This also makes the boot of the container pretty much instantaneous, as Docker doesn’t need to copy whole root filesystem.

This idea goes a bit further: the image itself is actually a frozen container. To prepare a new image, you just start a container, run one or more commands to install and prepare the application, and then commit the container as a new image. This means that your new image has only the files that have been added or changed since the image you started it from; and that base image has only files that have changed since its base image, and so on. At the very bottom there’s a base image that has been created from a filesystem archive – the only one that actually contains all of its files. There’s even a cool Dockerfile configuration that lets you describe how to build the application from the base image in one place. And this is where the layering goes a bit too far.

Two versions of the same Docker imageDocker itself is intentionally limited: when you start a container, you’re allowed to run only a single command, and that’s all. Then the container exits, and you can either dispose of it, or commit it as a new image. For running containers, it’s fine – it enforces clean design and separation of concerns. When building images, though, every “RUN” line means a new image is committed, which is a base for the next “RUN” line, and so on. When building any reasonably complex software based only on Dockerfile, we always and up with a whole stack of intermediate images that aren’t useful in any way. In fact, they are harmful, as there seems to be a limit on how many directories can you stack with aufs. It’s reported to be 42 layers, which is not too much considering size of some Dockerfiles floating around.

It seems that flattening existing images is not a simple task. Can we build images in one go, without stacking dozens of them on top of each other? It seems it’s actually quite easy. You just compress all the “RUN” Dockerfile statements into one shell script, and you’re halfway there. If you compress all the “ADD” statements into a single one too, you’re almost there: there’s one image for ADD, and a second one to RUN the setup script. Besides an unnecessary intermediate image being plain ugly, we have another issue: the ADD line often copies a big installer or package into an image, only to have it removed by a RUN line after installation. The user still has to download the intermediate image with the huge package file, only to not see it because child image had it deleted.

It turns out we can use shared volumes instead of the ADD statement. If we use docker run + docker commit manually rather than docker build with a Dockerfile, we can download all the installers on the builder host, expose it to the container as a shared volume, and then commit the container into an image in a single pass.

It’s possible to just write a shell script which does all of that manually. But the Dockerfile format is quite comfortable to use, and there’s quite a few container definitions already available that we wouldn’t be able to reuse. And it’s completely feasible to just read the Dockerfile syntax by a script that would execute it as a one-pass single-layer build:

  • Compose a single dict out of metadata commands, such as MAINTAINER, CMD, or ENTRYPOINT
  • For each RUN command, add a line to a setup shell script in a shared directory
  • For each ADD command, copy the data inside the shared directory, and add a line to the setup script that copies it to its final location
  • Build image in one pass, with a single docker run -v $shared_dir:/.data $from /.data/setup.sh, and if it’s successful, commit the image with a single docker commit using the metadata gathered earlier.

As it turns out, 125 lines of Perl is all it takes. The docker-compile.pl script is in the Gist below, and in the image to the right you can see inheritance diagram of the lopter/collectd-graphite image in two versions: on the right is the original, created by docker build; on the left is one created with docker-compile.pl. The flat one takes 299 MB; the combined ancestry of the original is almost 600MB.

All the script needs is Perl and JSON CPAN module (on Debian or Ubuntu, you can install it with sudo apt-get install libjson-perl). I hope the idea will prove useful – if there’s demand, I can take this proof of concept, polish it, document, and set up a proper GitHub repo with issue tracking and all. Happy hacking!

Access Chef node’s attributes by JSONPath

I have just released chef-helpers 0.0.7. The gem is just a bag of helpers for Chef recipes, without any common theme. This release adds one feature that may be really useful: it overloads Chef::Node#[] to allow deep indexing with JSONPath.

When is it useful? A code snippet is worth a thousand words.

Accessing nested attribute

Often I need to get value of a deeply nested attribute – which may not exist. If it doesn’t exist, its parent and grandparent may not be there as well. I need to code defensively (or resort to a blanket rescue clause) to take care of that:

Accessing by JSONPath makes it simpler:

Avoiding traversal when I don’t know the path upfront

When I generate random passwords, and I want to be chef-solo compatible, I want to be able to say something like:

This is supposed to raise an error when I run on chef-solo and I don’t have the attribute defined, and set them to a secure password by default when I have chef server to store it. The syntax may be a bit nicer and DRYer, but it’s already better than enumerating the attributes line by line. The implementation, though, is quite hairy – the inject method is powerful, but it’s not obvious what is done when you read the code:

With JSONPath, I can replace this line simply by:

Selecting attributes by value

I need to check whether a partition is mounted at a given path. The code used to look like this:

With JSONPath, I can focus on specifying what I expect to find, rather than how to traverse the nested hash:

I hope it will be useful – for sure it simplifies many similar pieces of code I needed to write in my recipes. Happy hacking!

Help MiniGit: need opinion on syntax

MiniGit is a small Ruby gem to talk with Git command line in a straightforward way. You can write, say, git.commit(:amend => true, :reuse_message => true), and have it do the right thing. No fancy model objects for repos, commits, and branches. No magic. Well, a little bit of magic, just to make it simple and comfortable to use.

One of the quirks is capturing Git’s output. As of now, MiniGit either lets git use the original console, or captures its whole stdout and returns it as a string. Without stripping the trailing newline, or any logic. Simple and straightforward – but pretty uncomfortable: to, say, make sure there are no changes to the work tree, you write git.capturing.diff.strip.empty?. It’s a handful, and don’t get me started on splitting the output into lines.

I want it to be simple. I want to be able to get the output as string – stripped or not, as I prefer at the moment. I want to be able to iterate over the lines, and don’t have the lines include the trailing newline. I want to be able to discard the output, or to display it only when there’s an error. And I don’t want to type too much to achieve that. But I have no good concept of how to write it.

A custom magic method (such as MiniGit#capturing) for each case would clutter the namespace and be hard to maintain. And it would involve a lot of output. The next idea – a custom MiniGit#[] method to get a customized instance – this syntax is reserved for git config integration: git['user.email'] is too pretty and intuitive to waste it on anything else.

Next thing I thought of was overloading unary operators: -git could discard output (like prefixing Make lines), ~git could capture stdout, etc. Two issues here: one is that there’s still a very limited number of these operators. Still, there’s a limited number of things I may want to do with Git’s output, so this may not even be a problem. Second issue is that method call binds closer than any operator: -git.push would mean -(git.push), and I’d need to add extra parentheses to say (-git).push – ugly! Not to mention it would require users to memorize what each of the symbols means.

The idea I’m thinking of right now may be short enough, but it requires a bit of magic, and I’m not sure if it’s nice & convenient magic, or testrocket flashy & useless magic. If I’d need to use parentheses anyway, I may override binary operators to mimic shell redirection syntax. Instead of magic capturing method, one could write:

Each of these expressions would create a new, specialized instance (in most cases cached in the parent instance to avoid allocating too many short lived instances, at cost of sometimes storing a few too many) just the way MiniGit#capturing and #noncapturing. Other modifiers could make it e.g. return a boolean variable to indicate success rather than always raising exception on error and requiring user to rescue when they need to check status.

Update: a more straightforward option would be a method that would be quicker to type, and accept either a known keyword, or any MiniGit subclass; something like this:

Write me what you think

What do you think: nice? uselessly convoluted? confusing? stupid? Or maybe you have any better ideas for that? I’ll happy to hear any input. You can also help by upvoting on Hacker News to get more people here.

Just use the poll here – it’s not directly published as a comment, but sent directly to me. I will update the article to include the results after a few days. If you’d like me to be able to reply, include your e-mail – but it’s not required in any way.

Waiting to hear from you!

Announcing knife-briefcase

One of the stumbling blocks for teamwork with Chef is sharing secrets; there are some files and other data that are sensitive, should not be committed to the version control repository, and the admins (or administration scripts) need them in daily work. Such files include secrets for encrypted data bags, private SSL keys, AWS keys (and other access keys and passwords), or private SSH keys to log into newly created EC2 instances. Until now, I kept these in .chef directory, and either had some crude script to pack them in a GPG-encrypted tarball stored somewhere (Dropbox, a file accessible via SSH, whatever), or was just sending around GPG-encrypted keys and files over e-mail.

Then I figured, hey, Chef server already has facilities to store and retrieve small pieces of data. Data bags don’t have to be used by recipes! What if I could just GPG-encrypt the files as I used to, but had a plugin to keep these in the Chef server?

This turns out to work quite nicely. The “knife briefcase” plugin (because you don’t store sensitive paperwork in a bag – you need a proper briefcase!) is available at https://github.com/3ofcoins/knife-briefcase and soon on Rubygems (I want to give it more real world testing first). It uses GPG (via ruby-gpgme) to encrypt and sign the content that is then stored in a data bag – and can be retrieved and decrypted. If list of authorized people changes (as you add new team member, or someobody leaves), knife briefcase reload will re-encrypt all the items to make the change easy. Simple, and effective.

I am aware that there’s already a plugin that serves similar purpose: chef-vault. I even use it in some of my cookbooks. But I don’t like it for this particular use case for two reasons.

First is minor: the command line requires me to always provide a node search query. If I want to share a secret only between admins, I need to provide a fake query that will return no results. It also requires some plumbing to re-encrypt the items. It’s not this much of an issue – it just requires some hacking. If it was the only reason, I’d happily send a pull request rather than write a separate new tool.

The second reason is that it’s complicated to set up a new person. Chef-vault uses administrator’s Chef API secret key to encrypt the data. This means that to re-encrypt the data for new team member, they need to create their API user first, and it requires a back-and-forth:

  1. New user gives me public SSH and GPG keys (which are needed anyway)
  2. I configure SSH access for the provided key and ping them to say they can start
  3. New user is able to create Chef API user now
  4. They ping me, or someone else who has access to the data, to re-encrypt it
  5. They are blocked until this is done

With knife-briefcase, the flow is much more async:

  1. New user gives me SSH and GPG public key (which they need to do anyway)
  2. I configure SSH access for the provided key, use GPG key to re-encrypt the data, and ping them to say they can start
  3. New user is able to create Chef API user and start work right away

This way, there’s one back-and-forth less, and the whole setup feels much simpler to the newbie, who can focus on getting to know the project and getting some work done rather than wait for access to be configured. Complicated setup is my pet peeve, and over last months I did quite a bit of work to simplify the agile administrator’s command center. I think I’m getting close to a nice setup – I should sit down and describe it in a separate blog post someday.

Meanwhile, have fun with knife-briefcase!

Installing Vagrant plugins from local files

tl;dr: to install a Vagrant plugin locally before publishing it to Rubygems, use bundle cache and create a local rubygems index.

I have just bumped into a weird problem: I’m creating an Omnibus package, and I want to run builders for different systems with Vagrant. That’s supported and should run out of the box; but life would be too easy if it worked that way. There were some Gem updates, some dependency mess I’m too lazy to untangle, and as I tried to start vagrant up, I ran into an obscure backtrace (and I wasn’t the only person that saw this error). Seems like a new problem, and there’s already an issue open for it on vagrant-berkshelf project.

Upgrade to Berkshelf 2.0. With a bit of luck, it should be a one-liner. Clone, bundle, run tests; upgrade Berkshelf in .gemspec, bundle, run tests. They pass. Nice. Let’s try with my installed Vagrant… now, how do I install the patched plugin?

Let’s see…

Oh, well. According to Vagrant docs, *the* supported way to install plugins is by publishing them on Rubygems. I don’t want to publish it yet, I’ve just created an untested patch!

The vagrant plugin install command can accept file name. I suppose the file should be a built gem (the documentation doesn’t mention that), so let’s try. Sweet, now it fails to fetch the gem’s dependencies.

There also is a --plugin-source PLUGIN_SOURCE option, documented as Add a RubyGems repository source. If I generate a rubygems index for my single gem, it still fails to find dependencies. Looks like it rather means Use specified RubyGems repository source. I need all dependencies in place, together with my built gem. Bundler had such an option, I think…


  • Build a .gem file with your plugin (for vagrant-berkshelf, it was thor gem:build).
  • Run bundle cache to have all dependencies as .gem files in vendor/cache directory.
  • Copy built pkg/vagrant-berkshelf-1.2.0.gem gem to vendor/cache
  • cd vendor/cache
  • Build a RubyGems repository index: gem generate_index
  • Install the plugin out of local RubyGems repository: /usr/bin/vagrant plugin install vagrant-berkshelf --plugin-source file://pwd

I needed to run /usr/bin/vagrant explicitly, because I let Bundler install Vagrant as a local Ruby gem and put it in my path before system-wide Vagrant. If you do bundle install --path=somewhere/else (I usually go with vendor/gems), you won’t need to do this.

I hope this will save someone else an hour or two of their life, or even a gray hair – maybe it will be myself a month from now, trying to remember what was the magic there!

Don’t name your config Toolfile (or .tool.yml)

There is a pattern, mostly used in Ruby world, that if you write a command line tool, named – say – The Tool, and it has a configuration file tied to a project you use it with, then the tool’s configuration fie is called Toolfile, and lives in the root of the project’s directory.

Don’t do that. Please.

I suppose that this started a couple centuries ago, with mother of all the build tools Make and its Makefile. Ruby build tool Rake used the existing convention for build tools, and named its configuration file Rakefile. This makes sense: in a software project, build script is the entry point. It’s logical and convenient that it lives in top-level directory, and it’s easy to locate, because the name is capitalized.

Rake followed a useful convention for build script. But then, all other tools followed Rake without stopping to think if it’s useful. Now I have a repository that has a Rakefile, Thorfile, Gemfile, Berksfile, Vagrantfile, and Strainerfile – there are more Toolfiles in there than real content.

The other custom is to name the config .tool.yml – also in project root. While Toolfiles clutter the directory by being too visible, dotfiles files do the opposite – they disappear. Some projects get this right and use dotfiles or dot-directories for stuff that should be hidden: .rvmrc, .bundle, or even .chef are good examples. Still, .travis.yml or .rspec are ugly. Even without the dot, cucumber.yml or Jekyll’s _config.yml smell quite bad.

How to name configuration files, then? Simple: use a config directory. Let me put all the config in the config/ subdirectory instead of cluttering the root and making it harder to spot the README file. Seriously, what would be wrong with config/thor.rb, config/gems.rb, config/berkshelf.rb, or config/travis.yml? It would be even possible to adapt to different projects’ conventions by using a dotfile (that is supposed to be hidden) pointing to the directory, and looking for the config like this:

  • If .config is an empty file, always use Toolfile and don’t look at any directory
  • If .config is a file, go with `cat .config`/tool.rb
  • If .config is a directory or symlink pointing to one, `realpath .config`/tool.rb
  • If there is no .config, but config/tool.rb exists, use it
  • At the end, try Toolfile

By default, tool would look for config/tool.rb or Toolfile, and any project that uses a different config directory could use .config file or symlink to override this. Vendorificator already partially implements that, and I think that I’ll soon release a small gem that would deal with config files, including the .config override. Some other options may be useful, like finding multiple config file fragments by directory or glob.
Let’s hope that in a year or two I’ll be able to move at least some of the configuration files away and won’t need to find my project’s files between the configs!