How to Add Space to Your Existing Mongodb Instance

We’ve Mongodb instance with RAID-10 array with Logical volumes. Just few days back we came to the point where we’ve to expand it or reshape the whole instance. Since we haven’t planned to get into mongodb replication, as our single node can serve our demand for the time being. We thought how about adding up few more disks.

So here are the steps we’ve followed to add new space in our current instance:

1. First Created 4 more EBS from AWS console (with 200GB + IOPS 600)
2. Kept Backup for Existing Drives (Can’t bet on uncertainty)
3. Attached 4 new EBS with Mongodb EC2 instance
4. Log into Mongodb Instance
5. Created New RAID-10 array
6. Created Physical Drive
7. Extended Volume Group
8. Allocated New Space
9. Extended Filesystem to Utilize New Space
10. Checked & Done!

First Created 4 more EBS from AWS console (with 200GB + IOPS 600)

Just created 4 more EBS drives from AWS Volumes page. We’ve set 200GB with IOPS 600. Then attached them with Mongodb EC2 Instance.

Kept Backup for Existing Drives (Can’t bet on uncertainty)

You can take snap from AWS console or you can do it using command line tools. Check out status over the AWS console, It take a while based on your storage size.

Attached 4 new EBS with Mongodb EC2 instance

Attach your newly created 4 EBS volumes and assign name which you can recall later. We used /dev/sdn1 to 4. But those were renamed to /dev/xvdn1-4 by system itself. Refresh AWS console just to make sure you’ve connected to all of them.

Log into Mongodb Instance

$ssh your-server-address

Created New RAID-10 array

We will name this new array /dev/md1 because our first array was named /dev/md0

Just execute the following command -

$ sudo mdadm --verbose --create /dev/md1 --level=10 --chunk=256 --raid-devices=4 /dev/xvdn1 /dev/xvdn2 /dev/xvdn3 /dev/xvdn4

So this should be done without any error (If all of your volumes are ok). You can execute the following code to see the detail -

$ sudo mdadm --detail /dev/md1

Add new array to mdadm configuration file -

$ echo 'DEVICE /dev/xvdn1 /dev/xvdn2 /dev/xvdn3 /dev/xvdn4' | sudo tee -a   /etc/mdadm/mdadm.conf
$ sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Created Physical Drive

Based on mongodb tutorial, this is how read-ahead is configured -

$ sudo blockdev --setra 128 /dev/md1
$ sudo blockdev --setra 128 /dev/xvdn1
$ sudo blockdev --setra 128 /dev/xvdn2
$ sudo blockdev --setra 128 /dev/xvdn3
$ sudo blockdev --setra 128 /dev/xvdn4

Set zero to first 512 bytes of first block.

$ sudo dd if=/dev/zero of=/dev/md0 bs=512 count=1

Now finally create physical drive -

$ sudo pvcreate /dev/md1

Extended Volume Group

We have new physical drive, now we can attach them with our existing volume group.

$ sudo vgextend vg0 /dev/md1

Allocated New Space

$ sudo vgs

Check how much unallocated (VFree) space is listed. In our case we got around 400GB space to allocate. We’ve to expand our existing partitions /data, /log and /journal.

You can use %FREE if you want to allocate by percentage or you can set specific number. We’ve done with % so let’s show we did.

$ sudo lvextend /dev/vg0/data -l +80%FREE
$ sudo lvextend /dev/vg0/journal -l +10%FREE
$ sudo lvextend /dev/vg0/log -l +10%FREE

And kept few space as unallocated so if we need in future we can utilize them.

Extended Filesystem to Utilize New Space

Unmount from system -

$ sudo umount /data

Checkout partitions if everything is ok -

$ sudo e2fsck -f /dev/vg0/data

Resize partition with the following command – (This should utilize all available space from the volume)

$ sudo resize2fs -p /dev/vg0/data

Continue this process for all of your partitions. After it’s done mount all devices.

$ sudo mount /data

Note: In case if you can’t unmount some device, You can try -

$ lsof | grep /log

And terminate process which is holding it back, Now try to unmount.

Checked! & Done!

Now let’s check how much storage you have. -

$ df -h

Thats all :)

Btw i’ve followed most instructions from - http://docs.mongodb.org/ecosystem/tutorial/install-mongodb-on-amazon-ec2/#install-mongodb-on-amazon-ec2 and later from here - http://pubmem.wordpress.com/2010/09/16/how-to-resize-lvm-logical-volumes-with-ext4-as-filesystem/

Thanks to them :)

Reposted in my personal blog - http://we4tech.wordpress.com/2013/07/19/how-to-add-space-to-your-existing-mongodb-instance/