Ohai Chefs!

At work, we’re being converted to the gospel of Etsy’s Church of Graphs. We’re sending Chef run times and other metrics to a combination of StatsD, Graphite, and . Last week, I wanted to add a graph of chef clients by version. In other words, I wanted to see how many Chef 0.10.8, and 10.12 clients we have left to upgrade.

This week, we’ll see how to get Chef client versions from the Chef Server, including one way which turned out to be more than 30 times faster in my tests.

The First Attempt

I needed to get a count of Chef clients grouped by version. I envisioned ending up with a hash like this:

1
2
3
4
5

  {
    '10.12.0' => 112,
    '10.16.2' => 534,
    '10.18.2' => 1
  }

My first thought was to do this:

1

$ knife node list | xargs -I {} knife node show {} -a chef_packages.chef.version -Fj

This pipes a list of all nodes in an org to knife node show and returns chef_packages.chef.version in JSON format.

This works, but it takes a loooong time, nearly 40 minutes on my quad-core Macbook Pro against our Private Chef server to get the Chef client version for 908 nodes, or ~2.4 seconds per node.

1
2
3
4
5

knife node list
  1.82s user 0.36s system 75% cpu 2.878 total

xargs -I {} knife node show {} -a chef_packages.chef.version
  1771.53s user 278.24s system 85% cpu ** 39:46.55 total **

This takes so long because knife node show makes a round-trip to the Chef server for each node. We need to speed this up, preferably by an order of magnitude.

The Second Attempt

What if we asked the Chef server to get Chef Client version info for every node in the org and send it to us in one batch?

1

knife search node 'name:*' -a chef_packages.chef.version –Fj

This approach is much more efficient; just over a minute instead of 40 minutes:

1
2

knife search node 'name:*' -a chef_packages.chef.version –Fj
  40.72s user 1.54s system 57% cpu ** 1:13.81 total **

The results from the knife search command look like this; easily parsable JSON.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

{
  "results": 3,
  "rows": [
    {
      "chef_packages.chef.version": "10.16.2",
      "id": "webserver01.example.com"
    },
    {
      "chef_packages.chef.version": "10.16.2",
      "id": "webserver02.example.com"
    },
    {
      "chef_packages.chef.version": "10.12.0",
      "id": "db01.example.com"
    }
  ]
}

Ohai Spelunking

But, hold on a second, how did I know the Chef client version attribute is named chef_packages.chef.version? I didn’t, but here’s how I found it:

1

knife node show myserver01.example.com -l | grep -C 5 10.16.2

I knew that myserver01.example.com was running Chef Client 10.16.2. I did a knife node show with the -l option to show all Ohai attributes and grep’d for 10.16.2 with five lines of context above and below (-C 5).

Here’s the result of that whole command:

1
2
3
4
5
6
7
8
9
10

Automatic Attributes (Ohai Data):
chef_packages:
  chef:
    chef_root:  C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.16.2/lib
    version:    10.16.2
  ohai:
    ohai_root:  C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.14.0/lib/ohai
    version:    6.14.0
command:           {}
counters:

From the output above, I can walk down the chef_packages attribute to determine the attribute I’m looking for is chef_packages.chef.version.

Parsing the Results

So, now that we have the raw JSON data, how can we turn it into this?

1
2
3
4
5

  {
    '10.12.0' => 112,
    '10.16.2' => 534,
    '10.18.2' => 1
  }

Let’s take a look at a script to parse the JSON list of nodes into a nice “grouped-by” version hash:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

require 'json'

KNIFE_RB = '.chef/knife.rb'
NODE_LIST = `knife search node -c #{ KNIFE_RB } 'name:*' -a chef_packages.chef.version --format json 2>&1`

def get_chef_clients_by_version(nodes)

  # turn JSON into Ruby objects
  nodes_json = JSON.parse nodes

  # create an array of all chef client versions
  client_versions = nodes_json['rows'].map { |item| item['chef_packages.chef.version'] }

  # initialize an empty hash to store our final counts grouped by version
  number_of_clients_by_version = Hash.new(0)

  # For each item in the client_versions array, create a unique key in our
  # number_of_clients_by_version hash and increment our counter
  client_versions.each { |version| number_of_clients_by_version[version] += 1 }

  number_of_clients_by_version
end

puts get_chef_clients_by_version(NODE_LIST)

Let’s take it line by line. On line one we’re requiring JSON. On line four we’re executing the knife search command. Lines 6 – 21 are a function to parse the node data into our final count of versions.

The get_chef_clients_by_version method

This method is where all the exciting stuff happens. On line 9, we’re parsing the JSON data and creating a Ruby data structure which looks like this:

1
2
3
4
5
6
7

{"results"=>3,
 "rows"=>
  [{"chef_packages.chef.version"=>"10.16.2", "id"=>"webserver01.example.com"},
   {"chef_packages.chef.version"=>"10.16.2", "id"=>"webserver02.example.com"},
   {"chef_packages.chef.version"=>"10.16.2", "id"=>"db01.example.com"},
  ]
}

Line 12 is my favorite line of the method.

1
2

  # create an array of all chef client versions
  client_versions = nodes_json['rows'].map { |item| item['chef_packages.chef.version'] }

It uses Ruby’s super useful map method to create an simple array of versions from the rows array of two-element hashes. The result of the map looks like this:

1

["10.16.2", "10.16.2", "10.12.0"]

From there we create and return the number_of_clients_by_version hash to hold our results and iterate over each item in the client_versions array, counting the nodes by version.

1
2
3
4
5
6

# initialize an empty hash to store our final counts grouped by version
  number_of_clients_by_version = Hash.new(0)

  # For each item in the client_versions array, create a unique key in our
  # number_of_clients_by_version hash and increment our counter
  client_versions.each { |version| number_of_clients_by_version[version] += 1 }

Thanks to my Talented and Gifted™ co-worker for this StackOverflow link which explained how to do the group-by version.

So here’s the result of the script, which is exactly what we set out to accomplish:

1
2
3
4
5

  {
    '10.12.0' => 112,
    '10.16.2' => 534,
    '10.18.2' => 1
  }

So there you have it. We compared two approaches to returning data from the Chef server, with one being an order of magnitude faster. We figured out how to find specific Ohai attribute names, and we created a script to transform the raw data to something truly useful.

These posts seem to keep getting longer, so maybe next week, we’ll have something short and sweet. Thanks for reading and I’d love to hear your comments.