Last Updated: July 19, 2018
· richthegeek

Tailable cursors in MongoDB

This took an hour or so of googling and reading the source of the Node driver to figure out, so I'm putting it here to save others the pain.

There's a neat feature of MongoDB wherein you can keep a cursor open on capped collection and waiting for data for the lifetime of the application. It's especially useful for creating a simple PubSub system without adding a dependency on Redis.

First you need to set up the capped collection:

db.createCollection('messages', {capped: true, size: 100000})

The "size" field is the maximum size of the collection in bytes. The above is 100kb, which is more than enough for the pubsub system (probably too much!)

Then, to get messages from that collection when they are inserted:

function listen(conditions, callback) {
  coll = db.collection('messages')
  latestCursor = coll.find(conditions).sort({$natural: -1}).limit(1)
  latestCursor.nextObject(function(err, latest) {
    if (latest) {
      conditions._id = {$gt: latest._id}
    options = {
      tailable: true,
      await_data: true,
      numberOfRetries: -1
    stream = coll.find(conditions, options).sort({$natural: -1}).stream()
    stream.on('data', callback)

The summary is as follows:
1. get the latest document and add it's ID to the conditions
2. create a cursor which is tailable, awaits data, and always retries
3. keep calling cursor.nextObject, passing results to the callback

The main thing that wasn't clear elsewhere is the option await_data, as it is usually written awaitdata.

Hope this helps someone!

2 Responses
Add your response

  1. It seems to actually be that the option is "awaitdata" nowadays, not "await_data" like it currently says in the post.
  2. It should be noted that the tailable cursor wont work if the collection is empty, so if you're unit testing, and clearing your database, you need to create an initial row or something before setting up the tail.
over 1 year ago ·

what do you mean "keep calling cursor.nextObject" ? i only see it called once here...

over 1 year ago ·