Hack 27. Pull Multiple Values from an Iterator

Make your iterators and generators highly context-sensitive.

Iterators and generators are fantastically useful for data that takes too long to generate, may never run out, or costs too much memory to keep around. Not every problem works when reading one item at a time though, and finding a nice syntax for pulling only as many items as you need can be tricky.

Perl's notion of context sets it apart from many other programming languages by doing what you mean. When you want a single item, it will give you a single item. When you want nothing, it can give you nothing. When you want a list, it will oblige. That power is yours through the wantarray( ) operator, too.

Wouldn't it be nice if Perl could tell how many items you want from an iterator or generator without you having to be explicit? Good newsit can.

Better Context than wantarray( )

Robin Houston's Want module extends and enhances wantarray( ) to give more details about the calling context of a function. Besides distinguishing between void and scalar context, Want's howmany( ) function can tell how many list items the calling context wants, from one to infinity.

The Code

Consider a simple generator that implements a counter. It takes the initial value, the destination value, and an optional step size (which defaults to 1). When it reaches the destination, it returns the undefined value.

sub counter {     my ($from, $to, $step)  = @_;     $step                 ||= 1;     return sub     {         return if $from > $to;         my $value       = $from;         $from          += $step;         return $value;     }; }

Creating and using a counter, perhaps one that counts from 1 to 10 by threes, is easy:

my $counter = counter( 1, 10, 3 ); my $first   = $counter->( );

What if you want the next three steps though? You could call it in a loop, but wouldn't it be nicer to call it with:

my ($first, $second, $third) = $iterator->( );

That's where multi_iterator( ) comes in. Feed it an iterator or generator and it returns a function that acts as a drop-in replacement for the iterator or generator but respects the calling context:

use Want 'howmany'; sub multi_iterator {     my ($iterator) = @_;     return sub     {         my $context = wantarray( );         return               unless defined $context;         return $iterator->( ) unless         $context;         return map { $iterator->( ) } 1 .. howmany( );     }; }

The multi-iterator first must check for void context (so it returns nothing and never kicks the contained iterator), then scalar context (so it can kick the iterator once). Then it kicks the iterator as many times as necessary to produce the number of expected values. Whatever the behavior of the contained iterator or generator when it exhausts its possible values, the multi-iterator will pass along to the caller.

This takes one more step than before, but the results speak for themselves:

my $counter          = counter( 1, 10, 3 ); my $iterator         = multi_iterator( $counter ); # multiple variables, list context my ($first, $second) = $iterator->( ); # void context $iterator->( ); # single variable, scalar context my $third            = $iterator->( ); # single variable, list context my ($fourth)         = $iterator->( );

$first contains the value 1 and $second the value 4. So far so good. $third contains 7 and $fourth 10. All subsequent accesses will contain undef.

Hacking the Hack

Being able to iterate over multiple iterators in parallel would be very useful. That's doable here.

This technique works outside of iterators as well; in any place you distinguish between list and scalar context and may need to know more about one-element list context versus n-element list context, howmany( ) is useful.

Want has many other interesting context-related features; it's worth exploring further on its own. Fortunately, its documentation is very useful.

Be careful about assigning the results of the iterator call to an array, which effectively has infinite elements. It may not do what you want if you have an infinite generator or iterator (unless you want an infinitely large array consuming infinite amounts of memory and taking infinite time to complete).