*pache

My current work involves making the infrastructure hosting the product much more scalable and flexible. A good step to have flexibility in the infrastructure is to have something quite flexible in front of it to proxy connections and possibly do something with them. At the moment our infrastructure provider handle the load balancing, and we can’t tweak things at all.

Here is how I got that layer in place.

What we wanted to achieve

First we want to be able to direct specific accounts (read : subdomains) to specific hosts if need be. It could be because they are so big they need to or because we want to run a little test with them for a specific feature.

Then we have also the need to be able to rollout an upgrade of the infrastructure in a smooth way : add a new host, route only a couple of accounts to it and see how it behaves.

All that is doable with many of the usual suspects (HA-Proxy, Nginx) but over a quick chat Jerome Petazzoni from Docker Inc. told me something in the essence of :

“why not Hipache ?”

Apache, for hipsters

There are 3 cousins in the french OpenSource community that are known to be serious ass kickers. One of them works for Docker Inc : Jerome. Over there he does tons of stuff including being part of the people writing Hipache “Apache but for hipsters”.

Needless to say, I do trust Jerome when it comes to those topics. So I’ve read a bit about Hipache, including a good article written by the main Hipache author and contributor : Sam Alba. You can find that article on the dot cloud blog : http://blog.dotcloud.com/under-the-hood-dotcloud-http-routing-layer.

In short : Hipache is aimed at being fast, reliable, cluster-able and use Redis to keep track of the endpoint configuration. Redis means that keeping several instances of Hipache in sync is super easy and super fast. I liked that idea.

Prepping

Hipache, like Docker, is under heavy development. Chances are the version you install on a machine will not play well with the configuration examples from the master branch. So do check the version your npm installs and do check the proper version of the documentation related to that tag in the repository. I did not initially do this and I’ve spent a couple of hours trying to figure out what was happening.

Hipache plays ok with HTTP and HTTPS you just have to pass the correct SSL certificates. So far, so good.

Hipache backends configuration happens in Redis with sets. Which means that you can easily do it by hand or, far more interesting, do it through you favourite language and any infrastructure management tool you are using.

Nginx, Unicorn

Our setup on each host involves Nginx and Unicorn. Nginx plays a role of micro caching and serving assets. Unicorn serves the Rails app. Our deploy and unicorn config allows us for zero downtime deployments. Nothing special here.

So far SSL connections stop at Nginx. We have a nice little block to handle the HTTP to HTTPS redirect with it :

server {
    listen 80;
    client_max_body_size 4G;
    return 301 https://$host$request_uri;
  }

Once Hipache in place HTTPS connections would stop with Hipache. The configuration does not cover that. And in fact, at the moment, Hipache is pretty dumb : input HTTP or HTTPS, output HTTP. And that’s it. You can’t tell Hipache “hey what ever comes on 80 you redirect to 443”.
If you think about it that makes sense : why rely on ports ? Hipache is an HTTP load balancer and proxy and that’s it.

I looked a bit around and then I remembered a short answer from Jerome :

“we check the headers for that, on the application side”

As I was insure of which headers he was talking about I searched and tweaked a little Rack app to see HTTP headers.

require 'rack'

class HttpHeaders
  def self.call(env)
    headers = env.select {|k,v| k.start_with? 'HTTP_'}
      .collect {|pair| [pair[0].sub(/^HTTP_/, ''), pair[1]]}
      .collect {|pair| pair.join(": ") << "<br>"}
      .sort
    [200, {'Content-Type' => 'text/html'}, headers]
  end
end

A little config.ru with it :

require 'rack'
require ‘./http_headers’

use Rack::SSL

app = Rack::Builder.new do
  run HttpHeaders
end

Rack::Server.start :app => app, :Port => 8082

I started the app, added the host as backend in Redis/Hipache and voila :

# HTTP
X_FORWARDED_PORT: 80
X_FORWARDED_PROTO: http
X_FORWARDED_PROTOCOL: http

# HTTPS
X_FORWARDED_PORT: 443
X_FORWARDED_PROTO: https
X_FORWARDED_PROTOCOL: https

The first one is what’s displayed if I connect to the host with HTTP, the second one if it’s with HTTPS.

I know that Rack middleware would easily be able to pick that up and respond fast without getting the rest of the app implicated. As Rack goes at least 2 gems do that already : Rack-SSL and Rack-SSL-enforcer.

A little bit of work in the config.ru :

require 'rubygems'
require 'rack'
require ‘./http_headers’
require 'rack/ssl'

use Rack::SSL

app = Rack::Builder.new do
  use Rack::SSL
  run HttpHeaders
end

Rack::Server.start :app => app, :Port => 8082

And well, it works.

So in the end here is the setup :

1. Hipache with a redis slave, HTTP and HTTPS configuration pointing towards a HTTP host
2. HTTP host (nginx) just connecting to unicorn through a socket
3. unicorn instance running the app with a rack middleware to request HTTPS as origin

Here is the link to Hipache github page : https://github.com/dotcloud/hipache and you can find an interview of Jerome Petazzoni on the Docker topic (http://blog.leanstack.io/how-docker-was-born ) that also contains bits about Hipache.