Of Containers, Dockers, Rockets, and Daemons

I have started using Docker soon after it showed up. I’ve been running dockerized services in production since well before 1.0. I am still using it quite extensively and recommend it to people.

I have also recently decided to run my next project on FreeBSD. I’ve been playing with this system for quite a while. It purportedly performs better than Linux — definitely feels faster and more reliable. It is engineered, not evolved, and base system is a consistent composition of kernel and user space. It’s been around for 21 years, more than any Linux distribution except Slackware. On top of that it has native ZFS and pf firewall. It’s quite refreshing to work with a system that… just works. Without fuss. Consistently. A system which is not a composition of tens and hundreds independent pieces of software, glued together with duct tape.

There’s just one “but”: Docker runs only on Linux, and I got used to containerized services. Jails are kind of similar, but the toolkits around them are like Linux’ LXC: you wrap entire system in a jail rather than a single service. I spent much time in the last weeks on a research/exploration effort: how much work would it take to run Docker on FreeBSD with jails for isolation, pf for networking, and zfs for storage? (There’s apparently some effort underway, but it seems stalled, and porting and maintaining the whole thing seems like a lot of effort). What would it take to implement some scripting that would quack in a Docker-like enough way to be useful? (Not much less, but I’d have freedom to be more opinionated about the mechanisms; on the other hand, list of features that would make it fully useful seems to converge to Docker’s documentation). All in all, to get things running in a reasonable time frame, I was going to suck it up, settle on nested jails with some ZFS-based tooling (of my own, no existing jail toolkit seems to fully utilize ZFS), and work on actual features. I already started to write some exploratory code.

At this very moment, CoreOS (any association with Oreo cookies purely accidental) announced that they are releasing their own container runtime, named Rocket. The announcement has since been edited (first version was much more aggressive). Docker responded (the text currently online is also edited, and much less defensive than the original). A quick flame war ensued, then passed, there was some name calling and mud slinging, business as usual on the Internet.

While I wouldn’t call Docker fundamentally flawed, the announcement, and my later exploration of Rocket, has shed some light on problems with Docker that I was vaguely aware of, but couldn’t really put my finger on:

  • Docker is implementation-defined. All specs seem to be an afterthought, and it is hard to predict which pieces of specification are stable. For example, Docker’s HTTP API is kind of documented (but not recommended: officially supported “API” is the command-line docker tool I recall having read the previous part, but cannot find it; the sentiments behind it are mirrored in #7538, or by the fact that there still doesn’t seem to be obvious way to express a docker run invocation through the API), but it leaks implementation details all over the place.
  • Docker is a monolyth. There is one binary that is both server and client, which is responsible for building the images, retrieving them from registry, managing volumes, layering filesystems, isolating processes in containers, managing containers as services (which makes it next to impossible to tie Dockerized services into your service manager without using a separate, idle process whose only role is to stay connected to Docker daemon and terminate as soon as the container exits; hello, unnecessary overhead!), configuring NAT, proxying network traffic, and probably a couple more things I didn’t think of.
  • Docker is opaque. I already wrote about HTTP API leaking implementation details, and being actively discouraged. The registry API is even worse. I have spent half an evening trying to figure out how to run a local, authenticated registry (and implement it if necessary). I gave up. Some of the API was documented; many details were buried in behaviour of the docker binary monolyth. It felt like Perl: kind of standarized and documented, but only perl can parse Perl. Seriously, what the hell? Why on Earth can’t downloading tar files work over plain HTTP(S)?

All of that gives a feeling of vendor lock-in and being on a whim of a company, which may or may not be benevolent, who has just received big investment and started to chase enterprise customers and partnerships. And current direction of Docker seems to point towards more complexity, not less. This direction makes all kind of sense for Docker, but it’s not one I really feel comfortable with — especially that until now application container tools were a Docker monoculture.

Enter Rocket. It is developed specification-first, and the implementation is officially a prototype. Specification focuses on one area only, and strictly distinguishes area of responsibility: building an image is separate from image discovery is separate from the image format is separate from container construction is separate from runtime environment. Each piece is independent, and can be used separately. Specification is clear and precise.

I wrote above about spending half an evening trying to figure out just the Docker registry api (together with relationship of registry vs index vs daemon vs client), and giving up. In a similar half-evening I went through Rocket’s app container specification (including image discovery, equivalent of Docker’s registry), was able to ask one of the developers a couple clarifying questions on IRC (CoreOS people were very friendly and helpful), and now I can say I understand how it works well enough to be able to implement it (given enough time, obviously), or reuse pieces in my own projects.

I don’t feel up to porting whole of Rocket to FreeBSD’s ports right now (but who knows? The codebase looks simple and straightforward), but as I am trying to quickly whip up something container-like on top of jails for my own project, I have some already written, well-designed specification I can work with, and some code that I can at least partially reuse. Docker is useless if I don’t port all of it; Rocket is useful even if I don’t use any of their code, and their code is modular enough for me to use only the pieces I want to. And while I am a bit sad that the whole thing began with name calling and mud slinging, and disappointed by Docker’s initial response, I am really happy to see some competition in the container management tools. Even if Docker stays dominant and Rocket doesn’t take off all the way, diversity of tools and preventing monoculture is valuable on its own. I have to admit: I’m pretty excited now.