8-mx6w
Last Updated: February 25, 2016
·
2.199K
· we4tech
Screen shot 2013 01 13 at 12.05.29 am

Ruby thread in simple rake task

An illustration how we can use ruby thread for accelerating our normal RAKE tasks. (intended for posting in a local Ruby developers group https://www.facebook.com/groups/rubydevs/)

namespace :somestuffs do
  desc 'Importing some big stuff that requies lotta network stuff'
  task :import
    file, tasks = ENV['FILE'], Queue.new

    # Parse out CSV file and retrieve each row and queue them up 
    # for further processing. 
    #
    # Keeping it a thread allows us to let this process continue while other 
    # threads are not in RUNNING state.
    Thread.new do
      CSV.open(file) { |row| tasks << row }
    end

    # Use something that matters in your envrionment, 
    # if you are using some network call which takes quite long time, 
    # then you can use 10/20 threads / CPU core
    4.times do
      Thread.new do
        while (row = tasks.pop)
          # Do network transfer related blocking task.
        end
      end
    end

    # Control thread which keeps current process blocked until queue becomes zero
    # In our case we are expecting tasks.size could get zero whenever no more data to process.
    Thread.new do
      sleep 1 until tasks.size.zero?
    end.join 

    # Calling "join" allows current process to wait until the JOINED thread is finished. 
    # Thus we can ensure our parent process doesn't exit until all tasks are done.
  end
end
Say Thanks
Respond