Last Updated: February 25, 2016
· we4tech

Ruby thread in simple rake task

An illustration how we can use ruby thread for accelerating our normal RAKE tasks. (intended for posting in a local Ruby developers group

namespace :somestuffs do
  desc 'Importing some big stuff that requies lotta network stuff'
  task :import
    file, tasks = ENV['FILE'],

    # Parse out CSV file and retrieve each row and queue them up 
    # for further processing. 
    # Keeping it a thread allows us to let this process continue while other 
    # threads are not in RUNNING state. do { |row| tasks << row }

    # Use something that matters in your envrionment, 
    # if you are using some network call which takes quite long time, 
    # then you can use 10/20 threads / CPU core
    4.times do do
        while (row = tasks.pop)
          # Do network transfer related blocking task.

    # Control thread which keeps current process blocked until queue becomes zero
    # In our case we are expecting tasks.size could get zero whenever no more data to process. do
      sleep 1 until

    # Calling "join" allows current process to wait until the JOINED thread is finished. 
    # Thus we can ensure our parent process doesn't exit until all tasks are done.