Spigot - A library for consuming third party APIs

I was working on a project recently, where I was associating data pulling from three independent services. The people that would be using my application had users and activity on each of them. My software was working out who was who and aggregating the data into a consolidated view.

As I was writing the clients for the different APIs, I noticed a common pattern emerging. Once I had received the data I needed, I transformed it into a shape I could use, and then dropped it into the model, usually with a find_or_create_by.

I decided to build a gem that would handle this using a simple yaml file in the config directory. Each file would be named after a service and in each service would be a set of resources.

This results in a config file such as:

# config/spigot/github.yml
user:
  full_name: name
  login: username
  contact: email

The key-value order was surprisingly important. I started off with the keys being the attribute name in your database. However, once you start considering api data that has nested hashes, it becomes difficult to properly convey the structure. Flipping it to mimic the structure of the received data, makes the config file format much more clear at a glance.

Spigot reads this file in and uses it to grab the values needed from the data and formats it into something your model can understand. The only other piece of info your model needs, is which service the data is coming from. Resulting in a method signature like this:

User.find_or_create_by_api(:github, data)

Nice and clean, just what I was looking for. After I finished writing the gem, I published it and included it in the original project that provided the reason for writing the gem in the first place. It felt great when I defined the config file, and replaced four methods on my user model, and it just worked on the first try.

Unfortunately, there are other problems that are arising, now that I'm getting into actual implementation. There are edge cases, such as preparing the data before, building associations after, interpolating values such as a gravatar_id into a url. These kinds of things should all be handled by your config file in the end. Hopefully after some more work, Spigot can take care of all these cases and help people clean up their API consumption.