Setting up ElasticSearch with MongoDB

I've recently set up an instance of ElasticSearch indexing data from MongoDB. I thought it would be helpful to others to put my notes online since I spent hours collating instructions from various sites.

Generally speaking, the ElasticSearch documentation sucks. It assumes a level of familiarity with search indexing out of the box that the average dev doesn't have. Setting up ES or Solr or Sphinx is very much a system administration chore, and configuring it is a discipline on its own.

The Environment

I used EC2 running an Unbuntu instance, so my notes are pretty specific to that environment. Wherever possible, I used the built-in package manager (apt/aptitude) and service manager (Upstart).

MongoDB

If you're not starting from scratch, you can skip this section as it pertains to installing Mongo.

To get Mongo up and running on Ubuntu, you will need to follow the excellent directions on 10gen's site.

You're going to be adding another upstream for apt, then pulling down and installing the mongod service into Upstart. Don't worry, apt takes care of doing all of this for you.

Don't try to skip this step. While the default Ubuntu apt package library contains Mongo, it's a really old version. I've had this bite me before.

Configure Mongo as a replica set

Even if you're running a standalone Mongo instance, you're going to need to convert it into a replica set. The reason is that the ElasticSearch plugin depends on the operation log (or 'oplog', a log of all changes used by Mongo to replicate itself) to push new updates into ElasticSearch. Mongo lacks built-in support for triggers, so this is the next best thing.

Again, follow the instructions here to convert your instance. However, ignore the line about starting it with the -replSet command line. Since we're running as a service, edit /etc/mongod.conf to include the replSet parameter. It doesn't matter what you name your replication set, just that you remember it and are consistent with it.

You can find more information about the Mongo configuration file here.

Installing ElasticSearch

I found this gist to be invaluable in getting the lastest version of ES up and running on my system. It'll first install a Java runtime, followed by ES and then a simple service wrapper.

After you've installed ES, and confirmed it's working by sending an HTTP request to localhost:9200, you have to install two plugins to enable support for MongoDB.

The first is a dependency called Mapper Attachments. You can install via the ES plugin script:

$ES_HOME/bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/1.6.0

$ES_HOME is wherever ES is installed. If you followed the instructions above, it'll be in /usr/local/share/elasticsearch.

The second plugin is the ES 'river' for Mongo. The syntax to install it is slightly different as it's a third-party plugin:

plugin -url https://github.com/downloads/richardwilly98/elasticsearch-river-mongodb/elasticsearch-river-mongodb-1.6.1.zip -install river-mongodb

Once those two plugins are successfully installed, you should be ready to go. Restart the ES service before proceeding to configuration just in case.

Configuring ElasticSearch

I don't want to go too deep into setting up your search index here. It's a very rich topic with lots of stuff specific to your data. The default analyzers that ES uses are pretty good anyway, so let's look at how you drive this thing.

You create and manage indexes in ES via a RESTful interface. The easiest way I've found is using curl inside of bash scripts to send JSON payloads which will instruct ES on how to look at our data.

Example:

#!/bin/sh
curl -XPUT "localhost:9200/_river/ffxi/_meta" -d '
{
  "type": "mongodb",
  "mongodb": {
    "servers": [
      { "host": "127.0.0.1", "port": 27017 }
    ],
    "options": { "secondary_read_preference": true },
    "db": "ffxi",
    "collection": "pages"
  },
  "index": {
    "name": "pagesidx",
    "type": "page"
  }
}'

Notice that the url contains _river. This is a small difference you'll notice when using 'river' plugins, that is non-native support for databases. This format is only for setting up and configuring an index however. When you query, it'll look more like:

curl -XGET 'localhost:9200/pagesidx/_search?q=param1