ActiveBlog

18 Months of docopt - A Polyglot CLI Option Parser
by Phil Whelan

Phil Whelan, April 3, 2014

--docopt=YES! Back in late 2012 when I was working on building kato, the system administration tool for Stackato, we were looking for the perfect option parser. Up to that point kato was written in Python, but for various reasons we decided to jump ship and write it from scratch in Ruby. The number of options that kato had was already getting overwhelming, so it was important to find a Ruby solution for managing command-line options that would scale.

As you can probably guess, the discussion on ActiveState's development list for "What's a good option parser?" was lively. Everyone had their favorites. Maybe they had used in the past. Maybe they had one that was "only supported language X, but language X is better than Ruby anyway :)" Each candidate parser seemed to have its caveats and short-comings.

Since we already had a set of options that kato supported, we wanted to continue to support this if possible. We did not want to rewrite the interface to kato based the constraints of a new option parser.

Then somebody recommended "docopt". Nobody had heard of it.

Inside Out

The first most interesting thing about docopt was that it worked in the opposite way to most of other option parsers. Instead of defining everything in code and having the code generate the usage documentation, the usage documentation is written first and this is the specification of the acceptable command-line arguments. I thought this was genius and it would definitely be maintainable at scale.

Here is an example usage file from kato.

Manipulate configuration values of Stackato components.

Usage:
  kato config get  [options] [<component>] [<key-path>]
  kato config set  [options]  <component>   <key-path>  [<value>]
  kato config del  [options]  <component>   <key-path>
  kato config push [options]  <component>   <key-path>  <value>
  kato config pop  [options]  <component>   <key-path>  <value>

Options:
  -h --help            Show help information
  -j --json            For "set", use JSON format when setting config key values.
                       For "get", use JSON format for displaying output.
  -y --yaml            Use YAML format when retrieving or setting config key values.
                       YAML is the default output format.
  -f --flat            Use a flat output format "<full-config-path> <value>"
  --force              Force updating value to different type.

Arguments:
  <value>              If value is not given for "set", then it read from STDIN.
  <component>          Can be "cluster", "local" or the name of a process.

Language Agnostic

When choosing an option parser for kato, I was in the process of porting kato from Python to Ruby. Therefore, the possibility that one day we might port kato to another language was not out of the realm of possibility. At that time docopt supported Ruby, Python, CoffeeScript, JavaScript, PHP, Bash, Lua and C. It now also supports C#/.NET and Go.

Support across multiple languages means that you can take your usage files and support the exact same interface in multiple languages. The parsing and error handling is all contained within the docopt implementation, so there is much less code to port over than there would be with other option parsers. Unfortunately, our Python version of kato had not been using docopt.

Command Hierarchy

The first thing you might notice is that the above "kato config" usage file is rather small and succinct. This usage file only covers one aspect of kato and we have built a hierarchal structure into kato, which is not natively supported with docopt.

We built a wrapper around our docopt usage, so that we can direct it at the appropriate usage file, if that usage file exists. This is done with a simple directory hierarchy and paired "usage" and "cmd.rb" files. The "usage" file is the docopt usage file and the "cmd.rb" implements the kato functionality for that part of the CLI.

kato's commands are generally structured as "kato ". In the example above, the noun being "config" and the verb being "get", "set" or "del" (delete) etc. It's not a strict rule, but it is a good guideline. Some commands fit a different structure, such as "kato " or "kato ". We want consistency, but not at the cost of our system administrators having to type long-winded commands for common actions. For instance, we chose "kato start" and "kato stop" over "kato node start" or "kato role start".

Here is a complete list of the current 60 usage files that make up kato's user interface.

phil@pmbp:~/src/activestate/kato/cli/lib/kato/cli/cmd (master)
$ find . -name usage
./config/usage
./data/export/usage
./data/import/usage
./data/usage
./data/users/usage
./debug/configwatch/usage
./debug/redis/usage
./debug/usage
./history/usage
./info/usage
./inspect/usage
./log/drain/add/usage
./log/drain/delete/usage
./log/drain/list/usage
./log/drain/status/usage
./log/drain/usage
./log/stream/usage
./log/tail/usage
./log/usage
./node/attach/usage
./node/availabilityzone/usage
./node/detach/usage
./node/list/usage
./node/migrate/usage
./node/placementzones/add/usage
./node/placementzones/list/usage
./node/placementzones/remove/usage
./node/placementzones/usage
./node/remove/usage
./node/rename/usage
./node/reset/usage
./node/retire/usage
./node/setup/core/usage
./node/setup/firstuser/usage
./node/setup/load_balancer/usage
./node/setup/micro/usage
./node/setup/usage
./node/update/usage
./node/usage
./node/version/usage
./op/usage
./patch/usage
./process/list/usage
./process/ready/usage
./process/restart/usage
./process/start/usage
./process/stop/usage
./process/usage
./relocate/usage
./report/usage
./restart/usage
./role/add/usage
./role/info/usage
./role/remove/usage
./role/usage
./shell/usage
./start/usage
./status/usage
./stop/usage
./version/usage

Some verbs, such as "kato role add" get their own usage file, because, like with our Ruby code, we want to limit the size of the usage file to something easily consumable. We do not want kato's help output to be too overly verbose. This would be the case if we listed the options of each verb and each verb took a lot of options.

If the user is only trying to use one specific action for a kato command, then they only see the options for that action, unless the total options for a command is quite small. Typing "kato role add --help" or "kato role add -h" or "kato role add help" or "kato help role add" will bring up just enough usage information for this specific command.

For some commands, such as the above "kato config", we have not yet felt the need to break out the verbs into their own usage files. If I type either "kato help config" or "kato help config set", I'll see the same "kato config" usage file output to my terminal.

So what happens if I type "kato help role" if there is a usage file for each verb? We put additional simple usage files higher up in the directory structure. These just list the commands. This could be optimized with a little bit of parsing of lower level usage files and auto-generation, but it has not become a maintenance issue yet. Also, we can write much clearer help text in the higher-level usage files, which gives the user a nice overview of what that area of kato does.

Where we do optimize is at the very top layer usage file. If you type simply "kato help" you will get a usage file that lists all the top level commands that kato supports. This is generated from an ERB template so, unlike the other static usage files, this does not port over to other languages as easily.

phil@pmbp:~/src/activestate/kato/cli/lib/kato/cli/cmd (master)
$ cat usage.erb
Stackato administration utility.

Usage:
  kato [command] [options]
  kato help [command]

Commands:

% cmds.sort_by { |name, description| name }.each do |name, description|
  <%= "%-10s" % name %>  <%= description %>
% end

kato fetches the name and first line description of any usage file it finds in the first level directory. It then prints them in name-sorted order.

Generating Documentation

Stackato's documentation is traditionally written in a markup language called "Sphinx". From here we generate the HTML documentation that you see at docs.stackato.com or bundled with the Stackato virtual appliance (go to /docs in your Stackato web console).

We wrote a tool to parse the all the usage files and generated kato reference documentation as Sphinx files. This pleased the documentation team immensely. They were able to integrate the kato Sphinx file generation into their own documentation build process. Now the documentation always 100% reflects the usage that kato accepts. This is true even if you are working on a git branch.

We still wanted the documentation team to be responsible for the documentation, so we gave them ownership of the usage files. They are responsible for fixing descriptions or commands and options. They were given clear guidelines about what changes would have what effect and how the docopt usage files were structured. So far, this collaboration around usage files has worked well.

Secondary Validation

Simple flags like --no-color need no extra intervention by the developer. They can simple use the boolean options[:no_color] in their code.

--no-color           Turn off color

This also applies to enumerated lists of options such as "(apple|banana|orange)". docopt will reject anything not fitting this clear list of possible values.

Where we need secondary validation, is where we accept values, such as node IP addresses.

-n --node <node-IP>  Only show logs from a specific cluster node

Many options in kato come up again and again. Therefore, we have plenty of help functions for the validation.

validate_node_ip(options[:node]) if options[:node]

Secondary Parsing

You will notice from the above examples that, for instance, where the option "-n --node " is given we get options[:node] in our Ruby code. Where we see --no-color, we get options[:no_color]. This is because kato normalizes the arguments that docopt returns. The output for docopt is fine for quick usage, but we wanted a little cleaner and consistent Ruby syntax if we were to use this across so much of our code. We prefer options[:no_color] over options['--no-color'], so our post-docopt options processor gives our Ruby this format.

A Simple Example

Here is the simplest example I could find in kato of a usage file with its relevant cmd.rb file. It will output the list of the log drains known to the Stackato cluster. The output will either be in YAML format, which is the default, or in JSON format.

phil@pmbp:~/src/activestate/kato/cli/lib/kato/cli/cmd (master)
$ ls -1 ./log/drain/list
cmd.rb
usage

usage file:

List all log drains

Usage:
  kato log drain list [options]

Options:
  -h --help       Show help information
  -y --yaml         Output at YAML
  -j --json         Output at JSON

cmd.rb file:

require 'kato/cli/opts'
require 'kato/logyard'

def kato_log_drain_list(argv)
  argv_to_options(__FILE__, argv) do |options|
    list = Kato::Logyard::list_drains
    if options[:json]
      output = Yajl::Encoder.encode(list, :pretty => true)
    else
      output = YAML.dump(list)
    end
    Kato::UI.puts output
  end
end

Interactive Shell

docopt.rb gave us a way to parse the usage files to some extent, but we needed to do a little extra work to expose all the details we needed in order to build "kato shell".

"kato shell" is an interactive version of the kato CLI. Simply type "kato" or "kato shell" on the command-line and it will drop you into the shell.

The shell provides tab completion and a way to see which commands are available based on what has been typed so far. It is much easier to hit the tab key to see what sub-commands are available, or typing "--" to see what options are available, than going back to the docs every time. It also helps with navigating through Stackato's cluster configuration.

Being an interpreted language, Ruby is not very fast when it comes to boot up time. We see over half a second overhead in starting up the kato Ruby process. With the kato shell everything is already loaded, so it makes things snappier when navigating around the commands and administrating your Stackato cluster.

Ruby

Stackato is built on-top of the Cloud Foundry open-source project. Ruby is still a large part of the Cloud Foundry code base, including the Cloud Controller. This was the major factor in choosing Ruby for kato. We wanted to expose some of kato's functionality through the Cloud Controller's REST API. Being able to drop a piece of kato's Ruby code into the Cloud Controller made sense and enabled us to keep things DRY.

Go

The Cloud Controller component of the Cloud Foundry project is still written in Ruby. I do not see this changing anytime soon, though things are moving quickly. We are seeing several other components receive Go love. Go seems to be the goto language for writing distributed systems these days and for good reason.

I was pleased to see a Go version of docopt appear on my radar last week, which is what led to this blog post.

Conclusion

For the past 18 months that we have used docopt, there have been few complaints. Only when we wanted to port a piece of functionality to kato, and the interface was non-standard, have we seen issue. I see this as a benefit. It ensures consistency across the user experience. docopt provides enough features to handle most situations, if approached in the right way.

My hope is that docopt will gain more adoption with newer command-line tools. I find docopt to be one of those clean separations of concerns, such as we have seen with code and view templates or HTML and CSS.

Learn more about Stackato


Ben Golub Explains Docker Inc

In July, Ben Golub joined as CEO. This enabled Solomon Hykes, founder and then CEO, to become CTO and truly focus on the technology.

Last week, I met with Ben Golub to find out more about Docker Inc and where they see things going.

Read more...

Alex Polvi Explains CoreOS

A couple of months ago we interviewed Solomon Hykes about Docker, which is a way to build and manage Linux Containers with a lot of nice features. The next question was: if the full-stack can be provided by a Docker image and everything can be Dockerized, what is the minimum OS we need to run Docker images? CoreOS was announced a few weeks ago and seemed to answer this question. For this blog post I interviewed Alex Polvi, the CEO of CoreOS, to find out more.

Read more...

Subscribe to ActiveState Blogs by Email

Share this post:

Category: stackato
About the Author: RSS

Phil Whelan has been a software developer at ActiveState since early 2012 and has been involved in many layers of the Stackato product, from the JavaScript-based web console right through to the Cloud Controller API. Phil has been the lead developer on kato, the command-line tool for administering Stackato. Phil's current role is a Technology Evangelist focused on Stackato. You will see Phil regularly on ActiveState's Blog. Prior to coming to ActiveState, Phil worked in London for BBC, helping build the iPlayer, and Cloudera in San Francisco, support Hadoop and HBase. He also spent time in Japan, where he worked for Livedoor.com and met his wife. Phil enjoys working with big data and has built several large-scale data processing applications including real-time search engines, log indexing and a global IP reputation network. You can find Phil on Twitter at @philwhln, where you can ask him any questions about Stackato. Alternatively, email at philw at activestate.com