For the last month, my team and I have been working on Gitbo, an app for finding open source software issues on Github and providing incentives for improvement by having users issue monetary bounties for solutions.
One of the challenges we had to overcome was managing our calls to the Github API. Github's starting limit on API requests for developers is 5000 per hour. Gitbo needs to pull all sorts of data from Github's API about developer's repositories, their related issues and comments, as well as user details.
Furthermore, we wanted to keep repository and issue information as fresh as we could in Gitbo. This required frequent calls to the Github API for every record in our database containing information from Github. It wasn't long before we started hitting the limit, which became a huge roadblock in development as well as a large red flag of future scalability problems for our app in production.
The Github API documentation outlines how to make conditional requests by adding an
If-Modified-Since header to the HTTP request to the API which will return a
304 Not Modified HTTP response if the resource hasn't been modified since that time. This was a great solution for our problem because any
304 responses from Github do not count against our rate limit. If the API resource has been modified since the time you requested, the API returns the normal JSON response.
The question was how to add this into our app. We had already implemented a fantastic open source API wrapper called
octokit, maintained by
pengwynn who has been a super helpful resource for our group. My fellow team members Adam and Kevin found a pull request to Octokit from
bratsche which adds support for making conditional requests. However, Wynn decided not to merge into Octokit, preferring a middleware solution, so they merged the new commits from
bratche/octokit into Adam's fork of
You can view the additional source code that adds the support in the merge commit.
If you'd like to make conditional requests in your application using
octokit simply add
octokit into your
Gemfile with Adam's repo as the source:
gem 'octokit', :github => 'ajonas04/octokit'
Make sure to
bundle install or
bundle update depending on whether you already have
Now with conditional requests, you can add a
:since option into all requests to Github that will make the request conditional, returning the full JSON response of the resource if its
Last-Modified header in the conditional request's response is later than the datetime query string you sent.
In order for it to work, you'll need to send the query string to Github in RFC 1123 Time Format. Luckily, Ruby's Time library has a method
httpdate which will convert a Ruby Datetime or Time object into a string with this format.
Let's see it in action, I'll make a conditional request with the current time as the
:since parameter. Since the resource will probably not have been modified in the time between the time object's creation and when Github receives the request, we'll see the benefits for our rate limit:
client = Octokit::Client.new(:login => username, :password => password) client.rate_limit_remaining #=> 5000 current_time = Time.now.httpdate client.repositories("joshrowley", :since => current_time) #=> nil # this nil tells us we received a 304 response client.rate_limit_remaining #=> 5000