Ruby thread in simple rake task
An illustration how we can use ruby thread for accelerating our normal RAKE tasks. (intended for posting in a local Ruby developers group https://www.facebook.com/groups/rubydevs/)
namespace :somestuffs do
desc 'Importing some big stuff that requies lotta network stuff'
task :import
file, tasks = ENV['FILE'], Queue.new
# Parse out CSV file and retrieve each row and queue them up
# for further processing.
#
# Keeping it a thread allows us to let this process continue while other
# threads are not in RUNNING state.
Thread.new do
CSV.open(file) { |row| tasks << row }
end
# Use something that matters in your envrionment,
# if you are using some network call which takes quite long time,
# then you can use 10/20 threads / CPU core
4.times do
Thread.new do
while (row = tasks.pop)
# Do network transfer related blocking task.
end
end
end
# Control thread which keeps current process blocked until queue becomes zero
# In our case we are expecting tasks.size could get zero whenever no more data to process.
Thread.new do
sleep 1 until tasks.size.zero?
end.join
# Calling "join" allows current process to wait until the JOINED thread is finished.
# Thus we can ensure our parent process doesn't exit until all tasks are done.
end
end
Written by nhm tanveer hossain khan (hasan)
Related protips
Have a fresh tip? Share with Coderwall community!
Post
Post a tip
Best
#Ruby
Authors
Sponsored by #native_company# — Learn More
#native_title#
#native_desc#