Autoscaling your Background Jobs on AWS

When you're developing an application, it's nice to be able to fire off a job to run in the background and not have to worry about whether or not it ran successfully. If you start sending more and more jobs, will your machines run out of disk space, run out of memory, alert you if a job failed? Using Iron's IronWorker product and your own AWS or GCP account, you can quickly set up a hands free background job processing system. The best part about these jobs is that they're run within containers so you can write them in any language.

In this example, we'll setup an autoscaling IronWorker cluster that will handle your jobs. The first step you'll need to do is signup at Iron.io and speak with their support about their "Hybrid" IronWorker clusters. Once your account is Hybrid enabled, you're ready to go.

Log into your Iron account and click on "My Clusters"

Now create a new cluster

Specify how many resources a job should have

If your job is sending email, it's probably less resource intensive and doesn't need that many resources available to it. If you're processing images or video however, that may be memory intensive, so you'll want to allocate more memory.

Once your cluster has been created, edit it and click "Add Autoscaling"

Notice this page also shows you how to start a job via Docker right from your command line! You'll mostly be using IronWorker's client libraries if your kicking off jobs from your application, but the command line approach is great for testing.

Add your Autoscaling parameters, click save, and you're done

Here you're able to add your cloud credentials, pick which machines you want to run your jobs on, and also specify the minimum and maximum amount of jobs you want running at any given time. If you have a machine with 10G of RAM available and each job can use 1G, you'll probably be safe with 9 jobs per machine. If you set the maximum job number to 15 for example, Iron will automatically fire up another instance on your end to run the extra jobs, then stop the instance when they're done.

Conclusion

With a few clicks, you set up infrastructure on your own cloud that can automatically handle scaling your background jobs. You can control the limits and Iron will take care of managing the machines for you. You can now fire off jobs via Iron's command line tool or from within your application with Iron's client libraries.