Overview

External iteration in Ruby operates through the Enumerator class and related methods that provide explicit control over iteration flow. Unlike internal iteration methods such as each and map, external iteration allows the caller to control when the next element is retrieved, enabling precise stepping through collections and infinite sequences.

The Enumerator class serves as the primary interface for external iteration. Ruby creates Enumerator objects automatically when calling iteration methods without blocks, or explicitly through Object#to_enum and Enumerator.new. These objects maintain iteration state and provide methods like next, peek, and rewind for manual traversal.

# Creating enumerators from existing collections
array_enum = [1, 2, 3].to_enum
string_enum = "hello".each_char

# Manual stepping through elements
puts array_enum.next  # => 1
puts array_enum.next  # => 2

External iteration supports lazy evaluation through Enumerator::Lazy, which chains operations without creating intermediate arrays. This approach handles large datasets and infinite sequences efficiently by computing values on demand.

# Lazy evaluation with external iteration
infinite_enum = (1..Float::INFINITY).lazy.map { |n| n * 2 }
first_five = infinite_enum.first(5)  # => [2, 4, 6, 8, 10]

The external iteration model integrates with Ruby's iteration protocol through StopIteration exceptions, which signal iteration completion. Ruby handles these exceptions internally in most contexts, but external iteration exposes them for fine-grained control.

Basic Usage

Creating Enumerator objects from existing collections uses to_enum or calls iteration methods without blocks. The to_enum method accepts method names and arguments, creating enumerators for any iteration method.

# Converting arrays to enumerators
numbers = [10, 20, 30, 40]
enum = numbers.to_enum

# Creating enumerators from strings
chars = "Ruby".each_char
lines = "line1\nline2\nline3".each_line

# Using method-specific enumerators
hash_enum = { a: 1, b: 2 }.each_pair

The next method retrieves the next element and advances the iteration position. Each call to next moves the internal pointer forward, maintaining state between calls.

enum = %w[apple banana cherry].to_enum

puts enum.next    # => "apple"
puts enum.next    # => "banana" 
puts enum.next    # => "cherry"
# Additional next call raises StopIteration

The peek method examines the next element without advancing the iteration position. This allows inspection of upcoming values while preserving the current state.

data = [100, 200, 300].to_enum

current = data.next     # => 100
upcoming = data.peek    # => 200 (doesn't advance)
actual_next = data.next # => 200 (now advances)

The rewind method resets the enumerator to its initial state, allowing complete re-iteration from the beginning.

enum = (1..3).to_enum

enum.next  # => 1
enum.next  # => 2
enum.rewind
enum.next  # => 1 (back to start)

Custom enumerators use Enumerator.new with blocks that yield values through the yielder parameter. This creates enumerators for custom iteration patterns.

fibonacci = Enumerator.new do |yielder|
  a, b = 0, 1
  loop do
    yielder << a
    a, b = b, a + b
  end
end

puts fibonacci.first(8)  # => [0, 1, 1, 2, 3, 5, 8, 13]

Advanced Usage

Chaining external iteration operations creates complex processing pipelines using Enumerator::Lazy. Lazy enumerators defer computation until values are actually needed, enabling efficient processing of large or infinite sequences.

# Complex lazy chain processing
result = (1..1_000_000)
  .lazy
  .select { |n| n.odd? }
  .map { |n| n * 2 }
  .take_while { |n| n < 1000 }
  .to_a

# Only processes necessary elements

Custom enumerators support stateful iteration by maintaining instance variables within the block scope. This enables complex iteration patterns with memory and conditional logic.

# Stateful enumerator with filtering
filtered_sequence = Enumerator.new do |yielder|
  @seen = Set.new
  @source = [1, 2, 2, 3, 1, 4, 3, 5]
  
  @source.each do |value|
    unless @seen.include?(value)
      @seen.add(value)
      yielder << value
    end
  end
end

puts filtered_sequence.to_a  # => [1, 2, 3, 4, 5]

Enumerators compose with other enumerators to create complex iteration hierarchies. Multiple enumerators can coordinate to produce interleaved or conditional sequences.

# Enumerator composition and coordination  
even_enum = (2..20).step(2).to_enum
odd_enum = (1..19).step(2).to_enum

interleaved = Enumerator.new do |yielder|
  loop do
    yielder << odd_enum.next
    yielder << even_enum.next
  end
end

puts interleaved.first(10)  # => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

External iteration supports custom indexing and position tracking through manual state management. Enumerators can maintain complex position information beyond simple linear advancement.

# Multi-dimensional iteration with position tracking
matrix_enum = Enumerator.new do |yielder|
  matrix = [[1, 2], [3, 4], [5, 6]]
  
  matrix.each_with_index do |row, row_idx|
    row.each_with_index do |value, col_idx|
      yielder << { value: value, position: [row_idx, col_idx] }
    end
  end
end

matrix_enum.each { |item| puts "#{item[:value]} at #{item[:position]}" }
# 1 at [0, 0]
# 2 at [0, 1]
# 3 at [1, 0]
# ...

Method chaining with external iteration combines multiple transformation steps while preserving lazy evaluation benefits. Each operation in the chain maintains the external iteration interface.

# Complex method chaining with external iteration
processed = File.readlines('data.txt')
  .lazy
  .map(&:strip)
  .reject(&:empty?)
  .map { |line| line.split(',') }
  .select { |fields| fields.length == 3 }
  .map { |fields| { id: fields[0], name: fields[1], value: fields[2].to_f } }

# Process only as many lines as needed
first_valid = processed.first(5)

Error Handling & Debugging

External iteration raises StopIteration exceptions when enumerators reach exhaustion. These exceptions carry the final return value in their result method, providing access to completion information.

enum = [1, 2, 3].to_enum

begin
  loop { puts enum.next }
rescue StopIteration => e
  puts "Iteration complete. Result: #{e.result}"
end
# Output: 1, 2, 3, then "Iteration complete. Result: [1, 2, 3]"

Debugging external iteration requires monitoring enumerator state and position. The current position is not directly accessible, but can be tracked manually or inferred through controlled iteration.

# Debugging enumerator state with position tracking
class DebuggingEnumerator
  def initialize(enum)
    @enum = enum
    @position = 0
  end
  
  def next
    value = @enum.next
    @position += 1
    puts "Position #{@position}: #{value}"
    value
  rescue StopIteration => e
    puts "Exhausted at position #{@position}"
    raise e
  end
  
  def peek
    @enum.peek
  end
end

debug_enum = DebuggingEnumerator.new([10, 20, 30].to_enum)
debug_enum.next  # Position 1: 10
debug_enum.next  # Position 2: 20

Invalid state access occurs when calling next or peek on exhausted enumerators. Proper error handling checks for StopIteration and provides fallback behavior.

def safe_next(enumerator, default = nil)
  enumerator.next
rescue StopIteration
  default
end

enum = [1, 2].to_enum
puts safe_next(enum)        # => 1
puts safe_next(enum)        # => 2  
puts safe_next(enum, :done) # => :done

Custom enumerators require careful exception handling within their definition blocks. Unhandled exceptions can corrupt the enumerator state or terminate iteration prematurely.

# Robust custom enumerator with error handling
safe_processor = Enumerator.new do |yielder|
  data = [1, 2, "invalid", 4, 5]
  
  data.each do |item|
    begin
      result = item * 2  # May raise exception for strings
      yielder << result
    rescue => e
      puts "Skipping invalid item: #{item} (#{e.message})"
      next  # Continue with next item
    end
  end
end

puts safe_processor.to_a  # => [2, 4, 8, 10] (skips "invalid")

Infinite enumerators require careful handling to prevent infinite loops in debugging and testing scenarios. Use first, take, or explicit limits when working with potentially infinite sequences.

# Safe handling of infinite enumerators
infinite_counter = Enumerator.new { |y| (1..Float::INFINITY).each { |i| y << i } }

# Dangerous: infinite_counter.to_a (never completes)
# Safe: 
sample = infinite_counter.first(100)
limited = infinite_counter.take(50)

Performance & Memory

Lazy evaluation with external iteration provides significant memory benefits for large datasets by avoiding intermediate array creation. Enumerator chains process elements one at a time rather than creating full intermediate collections.

# Memory-efficient processing of large datasets
large_data = (1..1_000_000)

# Memory-intensive approach (creates intermediate arrays)
result1 = large_data.select(&:odd?).map { |n| n * 2 }.first(100)

# Memory-efficient approach (lazy evaluation)
result2 = large_data.lazy.select(&:odd?).map { |n| n * 2 }.first(100)

# result2 uses constant memory regardless of large_data size

External iteration performance depends on the complexity of the underlying iteration method and the frequency of state access. Simple enumerators perform comparably to internal iteration, while complex stateful enumerators may have overhead.

require 'benchmark'

data = (1..100_000).to_a

Benchmark.bm do |x|
  x.report("Internal each:") { data.each { |n| n * 2 } }
  x.report("External enum:") { 
    enum = data.to_enum
    while true
      begin
        enum.next * 2
      rescue StopIteration
        break
      end
    end
  }
end

Custom enumerators with complex logic may have performance implications due to block execution overhead and state management. Profile custom enumerators to identify bottlenecks in computation-intensive scenarios.

# Performance comparison of enumerator implementations
def simple_range_enum(limit)
  (1..limit).to_enum
end

def custom_range_enum(limit)
  Enumerator.new do |yielder|
    (1..limit).each { |i| yielder << i }
  end
end

# simple_range_enum typically performs better due to optimized Range iteration

Memory usage in external iteration depends on enumerator chain complexity and intermediate state storage. Lazy enumerators maintain minimal state, while stateful enumerators may accumulate memory over time.

# Memory-conscious enumerator design
class MemoryEfficientProcessor
  def self.process_file(filename)
    Enumerator.new do |yielder|
      File.foreach(filename) do |line|
        # Process line immediately, don't store
        processed = line.strip.upcase
        yielder << processed if processed.length > 0
        # Line is garbage collected after processing
      end
    end
  end
end

# Processes large files with constant memory usage
processor = MemoryEfficientProcessor.process_file('large_data.txt')
processor.first(1000)  # Only loads necessary lines

Reference

Core Enumerator Methods

Enumerator Creation Methods

Enumerator::Lazy Methods

Exception Handling

Enumerator States

Common Patterns

# Pattern: Safe iteration with default values
def safe_iterate(enum, default_value)
  loop { yield enum.next }
rescue StopIteration
  default_value
end

# Pattern: Peeking ahead for conditional processing  
while enum.peek != :terminator rescue false
  process(enum.next)
end

# Pattern: Parallel iteration of multiple enumerators
def zip_enumerators(*enums)
  Enumerator.new do |yielder|
    loop do
      values = enums.map(&:next)
      yielder << values
    end
  end
rescue StopIteration
  # Ends when any enumerator exhausted
end

# Pattern: Stateful enumerator with cleanup
def resource_enumerator(resource)
  Enumerator.new do |yielder|
    begin
      resource.open
      resource.each { |item| yielder << item }
    ensure
      resource.close
    end
  end
end