Restoring a Compose MongoDB 3.0 backup on Heroku

Since March 2015, Compose has supported MongoDB 3.0. One of the nicer features of Compose is its automatic backups. However, when I needed to restore from a backup the other day, it took some back-and-forth with Compose's support team to resolve, and there were a few surprises along the way.

My hope is that this article will be outdated soon, but here's what you need to know today.

The short version

As recommended by the Compose support team:

1. Install MongoDB 2.6 on your local machine
2. Download and extract the backup
3. Run mongodump --dbpath path/to/extracted/dir
4. Run mongorestore using your Heroku credentials on the newly created dump directory

The rest of this article will go into more detail. But first, some key information:

Backups aren't 3.0-friendly

Each of these downloads is in a format that's not easy to import into MongoDB 3.0.

Why MongoDB 2.6?

I was surprised when Compose's support team asked if I could install MongoDB 2.6.

The short version is, you need to produce a set of BSON/JSON files from your backup in order to restore it. Support for this was removed in 2.8:

The 2.8 versions of MongoDB tools, mongodump, mongorestore, mongoexport, mongoimport, mongofiles, and mongooplog, must connect to running MongoDB instances and cannot modify MongoDB data files (i.e. with --dbpath) as in previous versions.

1. Install MongoDB 2.6

I'm using Homebrew on OS X. I found it easiest to uninstall mongodb and install it using homebrew-versions.

Important: Make sure you back up before you do this.

Install Homebrew versions, if you don't already have it:

brew tap homebrew/versions

Uninstall your local MongoDB:

brew uninstall mongodb

Install MongoDB 2.6:

brew install homebrew/versions/mongodb26

(Once you're finished, you can run brew uninstall homebrew/versions/mongodb26 and re-run brew install mongodb.)

2. Download and extract the backup

Extract tarball. In my case, I ended up with a directory that looked like this:

.
├── admin.0
├── admin.ns
├── app12345678.0
├── app12345678.1
├── app12345678.2
├── app12345678.3
├── app12345678.4
├── app12345678.ns
├── local.0
├── local.1
├── local.2
├── local.3
├── local.4
├── local.ns
├── mongod.lock
├── raw_parse.txt
└── storage.bson

3. Run mongodump

Inside the directory above, run:

mongodump --dbpath .

Note: the --dbpath option is only supported in MongoDB 2.6, hence why we had to install it.

This will produce a dump directory containing the data:

dump
├── admin
│   ├── system.indexes.bson
│   ├── system.users.bson
│   ├── system.users.metadata.json
│   ├── system.version.bson
│   └── system.version.metadata.json
└── app12345678
    ├── my_collection_name.bson
    ├── my_collection_name.metadata.json
    └── system.indexes.bson

4. Run mongorestore

You'll need your Heroku app's MONGOHQ_URL environment variable for the credentials:

heroku config:get MONGOHQ_URL

You should get something like this:

mongodb://heroku:averylongstring@candidate.2.mongolayer.com:5678,candidate.1.mongolayer.com:12345/app12345678

Note that there's two hosts and ports listed. In my case, the right one was the one listed second, but you can double-check by looking at the Admin section of the Compose panel (look for “Mongo Console”).

Your mongorestore should look like this:

mongorestore \
  --host candidate.1.mongolayer.com \
  --port 12345 \
  --db app12345678 \
  -u heroku \
  -p averylongstring \
  dump/app12345678/

That will restore all the files in the dump/app12345678 directory created by mongodump in the step above.

That's it!

Why the rigamarole?

From the support team:

The reason for this change is MongoDB's move to pluggable storage systems. We haven't yet decided how best to package backups for these scenarios: with any backup you download from us, you can use the extracted files directly to stand up a mongo instance locally, against which you can then perform mongodump/mongorestore operations. For your particular case, it's easiest to use pre-2.8 utils to create the type of files that mongorestore can work with.

Why isn't there a “restore” button next to each backup?

I suggested this along the way, and got this response:

We actually do have an automagic restoration button for each and every backup taken on a deployment... but only for deployments that are provisioned through a direct Compose.io account. The TL;DR on why that has to be is because of how the Heroku partner program integration works. There just wasn't a reasonable and reliable way that we could bring that same data restoration functionality to Heroku-provisioned databases. Sadness all around. For heroku databases, using standard mongodump / mongorestore tools is the only way to restore a backup.