We’ve Mongodb instance with RAID-10 array with Logical volumes. Just few days back we came to the point where we’ve to expand it or reshape the whole instance. Since we haven’t planned to get into mongodb replication, as our single node can serve our demand for the time being. We thought how about adding up few more disks.
So here are the steps we’ve followed to add new space in our current instance:
- First Created 4 more EBS from AWS console (with 200GB + IOPS 600)
- Kept Backup for Existing Drives (Can’t bet on uncertainty)
- Attached 4 new EBS with Mongodb EC2 instance
- Log into Mongodb Instance
- Created New RAID-10 array
- Created Physical Drive
- Extended Volume Group
- Allocated New Space
- Extended Filesystem to Utilize New Space
- Checked & Done!
First Created 4 more EBS from AWS console (with 200GB + IOPS 600)
Just created 4 more EBS drives from AWS Volumes page. We’ve set 200GB with IOPS 600. Then attached them with Mongodb EC2 Instance.
Kept Backup for Existing Drives (Can’t bet on uncertainty)
You can take snap from AWS console or you can do it using command line tools. Check out status over the AWS console, It take a while based on your storage size.
Attached 4 new EBS with Mongodb EC2 instance
Attach your newly created 4 EBS volumes and assign name which you can recall later. We used /dev/sdn1 to 4. But those were renamed to /dev/xvdn1-4 by system itself. Refresh AWS console just to make sure you’ve connected to all of them.
Log into Mongodb Instance
Created New RAID-10 array
We will name this new array /dev/md1 because our first array was named /dev/md0
Just execute the following command -
$ sudo mdadm --verbose --create /dev/md1 --level=10 --chunk=256 --raid-devices=4 /dev/xvdn1 /dev/xvdn2 /dev/xvdn3 /dev/xvdn4
So this should be done without any error (If all of your volumes are ok). You can execute the following code to see the detail -
$ sudo mdadm --detail /dev/md1
Add new array to mdadm configuration file -
$ echo 'DEVICE /dev/xvdn1 /dev/xvdn2 /dev/xvdn3 /dev/xvdn4' | sudo tee -a /etc/mdadm/mdadm.conf $ sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
Created Physical Drive
Based on mongodb tutorial, this is how read-ahead is configured -
$ sudo blockdev --setra 128 /dev/md1 $ sudo blockdev --setra 128 /dev/xvdn1 $ sudo blockdev --setra 128 /dev/xvdn2 $ sudo blockdev --setra 128 /dev/xvdn3 $ sudo blockdev --setra 128 /dev/xvdn4
Set zero to first 512 bytes of first block.
$ sudo dd if=/dev/zero of=/dev/md0 bs=512 count=1
Now finally create physical drive -
$ sudo pvcreate /dev/md1
Extended Volume Group
We have new physical drive, now we can attach them with our existing volume group.
$ sudo vgextend vg0 /dev/md1
Allocated New Space
$ sudo vgs
Check how much unallocated (VFree) space is listed. In our case we got around 400GB space to allocate. We’ve to expand our existing partitions /data, /log and /journal.
You can use %FREE if you want to allocate by percentage or you can set specific number. We’ve done with % so let’s show we did.
$ sudo lvextend /dev/vg0/data -l +80%FREE $ sudo lvextend /dev/vg0/journal -l +10%FREE $ sudo lvextend /dev/vg0/log -l +10%FREE
And kept few space as unallocated so if we need in future we can utilize them.
Extended Filesystem to Utilize New Space
Unmount from system -
$ sudo umount /data
Checkout partitions if everything is ok -
$ sudo e2fsck -f /dev/vg0/data
Resize partition with the following command – (This should utilize all available space from the volume)
$ sudo resize2fs -p /dev/vg0/data
Continue this process for all of your partitions. After it’s done mount all devices.
$ sudo mount /data
Note: In case if you can’t unmount some device, You can try -
$ lsof | grep /log
And terminate process which is holding it back, Now try to unmount.
Checked! & Done!
Now let’s check how much storage you have. -
$ df -h
Thats all :)
Btw i’ve followed most instructions from - http://docs.mongodb.org/ecosystem/tutorial/install-mongodb-on-amazon-ec2/#install-mongodb-on-amazon-ec2 and later from here - http://pubmem.wordpress.com/2010/09/16/how-to-resize-lvm-logical-volumes-with-ext4-as-filesystem/
Thanks to them :)
Reposted in my personal blog - http://we4tech.wordpress.com/2013/07/19/how-to-add-space-to-your-existing-mongodb-instance/