Last Updated: February 25, 2016
·
1.963K
· NIA

Making Ruby's Enumerator lazier

Problem:
Enumerable#map and its brothers return array. Ruby 1.9's Enumerator mixes in Enumerable. That is, calling #map, #find_all, #reject etc on an Enumerator will try to pull all items from it to gather them into array. Good, industrious Enumerator. But not lazy.

Why it is bad:
Consider an continual enumerator, having an infinite loop inside. For example, such a way to continually prompt user for something from console (and yield the results):

prompter = Enumerator.new{|y| loop{ y << gets } } 

It's easy now to make #each on this enumerator, but if we try to prepend it with some transformations...

prompter.map(&:strip).reject(&:empty?).each{|s| ... }

...we will fail to build command processor on it. Execution wil NOT reach the block of each until the prompter runs out, that is NEVER. That is the direct result of the Problem stated above.

How to fix:
If standard #map, #reject methods do not satisfy β€” let's write our own! A good idea how to do it is given here. I just give it here with some further extensions.

By adding a very general defer method we achieve full control over enumerator:

class Enumerator
  def defer(&blk)
    self.class.new do |y|
      each do |*input|
        blk.call(y, *input)
      end
    end
  end
end

The defer method returns a new Enumerator, passing its yielder and items into block.

So now we can work with it like this:

prompter.defer {|out, in| out << in.strip } # "map"
prompter.defer {|out, in| out << in if ... } # "reject"

Looks verbose... Yep, why not make shortcuts for it?

class Enumerator
  def lazy_map
    defer { |out, in| out << yield(in) }
  end
  def lazy_reject
    defer { |out, in| out << inp unless yield(inp) }
  end
end

Good! But in the fight against verbosity, we must be ultimate. So let's get back to the first listing and wrap it into a method, too, to make it look just like Scala's Iterator.continually:

class Enumerator
  def self.continually(&blk)
    self.new do |y|
      loop { y << blk.call }
    end
  end
end

Now we can finally write the prompter example in a short and readable way:

Enumerator.continually { gets }.
  lazy_map(&:strip).
  lazy_reject(&:empty?).
  each{|input| work_hard(input) }

Woohoo!

How to make it even better:
To carry the work through it would be great to:

  • find all the methods of Enumerable that can be made lazy;
  • instead of patching Enumerator with lazy... shit, create a subclass, in which override all these methods, and patch Enumerator only with tolazy method returning that subclass;
  • create a gem for this all.

Maybe I will try to do this when I have more spare time.

Thanks for reading! Hope this helps you to work with Enumerators with joy.

2 Responses
Add your response

FYI: In Ruby 2.0+ there's Enumerable#lazy. http://patshaughnessy.net/2013/4/3/ruby-2-0-works-hard-so-you-can-be-lazy

(Yes, I do realize this post talks about 1.9 πŸ˜‰)

over 1 year ago ·

@ravicious Thanks, that's important to know that the desired functionality has finally come official :)

over 1 year ago ·