Provision a StatsD server on EC2 using Vagrant and Chef

Its actually super easy to do. Vagrant 1.2.x allows for the aws provider, meaning Vagrant can provision an EC2 instance the same way it does a VM. Plus, if we use Berkshelf for cookbook dependency management, we can plug it right into the Vagrant process and avoid storing all those cookbooks locally.

./Berksfile

site :opscode
cookbook 'apt'
cookbook 'statsd', git: 'https://github.com/librato/statsd-cookbook.git'
cookbook 'graphite', git: 'https://github.com/hw-cookbooks/graphite.git'

./Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.omnibus.chef_version = :latest
  config.vm.box = "dummy"
  config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
  # Provider
  config.vm.provider :aws do |aws, override|
    aws.access_key_id = "<redacted>"
    aws.secret_access_key = "<redacted>"
    aws.keypair_name = "<redacted>"
    aws.ami = "ami-5f2abc6f" # "Ubuntu 12.04 LTS with Chef"
    aws.region = "us-west-2"
    aws.instance_type = "t1.micro"
    aws.security_groups = ["Basic", "StatsD"]
    override.ssh.username = "ubuntu"
    override.ssh.private_key_path = "path/to/your/key"
    aws.tags = {
      'Name' => 'Stats (Vagrant Provision)'
    }
  end
  # Provisioning
  config.berkshelf.enabled = true
  config.vm.provision :chef_solo do |chef|
    chef.add_recipe "apt"
    chef.add_recipe "statsd"
    chef.add_recipe "graphite"
  end
end

Only 2 files required! You'll want to make sure you have the following dependencies resolved:

• Vagrant 1.2.x installed
• berkshelf gem (gem install berkshelf)
• vagrant-berkshelf plugin installed (vagrant plugin install vagrant-berkshelf)
• vagrant-omnibus plugin installed (vagrant plugin install vagrant-omnibus)
• AWS access key and secret access key
• AWS local keypair
• EC2 Security group(s) allowing for
  • http (80) for the graphite web view
  • udp (8125 by default) the protocol and port used by statsd
  • ssh (22) required in order for vagrant to provision the instance

Assuming all those dependencies are covered, and the correct values are plugged into the Vagrantfile (such as access keys and key paths, as well as region and ami name if those need to be different), all it takes is the following command:

vagrant up --provider aws

That command will create the instance. Before provisioning, the omnibus plugin will make sure chef is installed, and the berskshelf plugin will run its install command, and the chef provisioning will handle the rest from there. Creating the instance and provisioning it can take a while, so be prepared to wait ~10 minutes.

Once its complete, open your ec2 instance in a web browser in a browser, and you'll find the Graphite interface. Send some stats to the StatsD server:

shell:

gem install statsd-ruby
irb

ruby:

require 'statsd-ruby'
statsd = Statsd.new "<your ec2 instance host name>"
# Incrementer:
statsd.increment "hello.world"
# Timer
statsd.timing 'glork', 320
# Guage
statsd.gauge 'bork', 100

Remember also that StatsD has additional backend options, and the statsd cookbook declared in the Berksfile here is maintained by Librato, which provides a backend of its own. Configuration is as simple as a few json attributes in the Vagrantfile.

I've also gone through this process in a gist here: https://gist.github.com/amoslanka/6245043

Caveats

The AWS world is a bit particular, and if something goes wrong you'll often get back a 400 response, which Vagrant is unable to explain beyond that, so there's not a lot of explanation. A few things I've noticed are as follows:

• Make sure the ami specified lives in the region specified.
• Make sure you specify security groups by name, not by id.