Property-Based Testing

Overview

Property-based testing validates software by checking whether properties hold true across a wide range of automatically generated test cases. Unlike example-based testing, which verifies behavior against predetermined inputs and outputs, property-based testing defines invariant characteristics that must remain true regardless of input values.

The approach originated with QuickCheck in Haskell, introduced by Koen Claessen and John Hughes in 2000. The core concept involves specifying properties as universal statements—assertions that should hold for all valid inputs—and then generating hundreds or thousands of random test cases to verify those properties.

A property-based test consists of three components: a generator that produces random input values, a property assertion that must hold true, and a shrinking mechanism that reduces failing cases to minimal reproducible examples. When a property fails, the testing framework automatically simplifies the failing input to the smallest value that still triggers the failure, accelerating debugging.

Consider testing a sorting function with example-based tests:

def test_sort
  assert_equal [1, 2, 3], sort([3, 1, 2])
  assert_equal [1, 5, 9], sort([9, 1, 5])
end

The same validation using properties:

property "sorted array is ordered" do
  array = generate_array
  sorted = sort(array)
  
  sorted.each_cons(2).all? { |a, b| a <= b }
end

The property-based version generates thousands of random arrays and verifies the sorting property holds for all of them. This discovers edge cases that manual example selection might miss: empty arrays, single elements, duplicates, negative numbers, very large arrays, already sorted inputs.

Property-based testing excels at finding boundary conditions and unexpected interactions. A single property can replace dozens of example-based tests while providing stronger guarantees about correctness. The approach requires different thinking—identifying what should always be true rather than what specific outputs match specific inputs.

Key Principles

Property-based testing centers on universal quantification: assertions that hold for all inputs in a domain. A property represents an invariant relationship or characteristic that remains constant regardless of which valid inputs the test uses.

Generators produce random test data within specified constraints. A generator for integers might produce values from negative billions to positive billions, while a generator for email addresses produces strings matching email format requirements. Generators compose: a generator for user objects might combine generators for strings, integers, and booleans.

# Simple generator
def generate_positive_int
  rand(1..1000)
end

# Composite generator
def generate_user
  {
    name: generate_string(1..50),
    age: generate_int(0..120),
    email: generate_email
  }
end

Shrinking transforms failing test cases into minimal examples. When a property fails for input [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], shrinking reduces it to the smallest failing case, perhaps [1, 2]. This occurs automatically through systematic input reduction: removing elements from collections, decreasing numeric values toward zero, shortening strings.

The shrinking process follows a search strategy. For arrays, remove elements from the end, then the middle, then the beginning. For numbers, try zero, then halfway between zero and the current value. For nested structures, shrink inner values before outer structure. The framework continues shrinking until no smaller input triggers the failure.

Property categories define common patterns for expressing invariants:

Invariant properties assert that certain characteristics remain unchanged by operations. Testing serialization: deserialize(serialize(x)) == x. Testing sorting: all elements in the output exist in the input.

Oracle properties compare implementation results against a trusted reference. Testing a new sorting algorithm: new_sort(array) == trusted_sort(array). Testing arithmetic operations against a calculator library.

Metamorphic properties relate multiple executions of the same function. Testing sort: sort(sort(x)) == sort(x). Testing set operations: union(a, b) == union(b, a).

Induction properties verify behavior on small cases generalizes to large ones. Testing list operations: reverse([]) == [] and reverse([x]) == [x] and reverse(x + y) == reverse(y) + reverse(x).

Test case generation uses configurable parameters. Run count determines how many random inputs to test—typically 100 to 1000 per property. Size parameters control the magnitude of generated values: string lengths, array sizes, numeric ranges. Seed values enable reproducible random generation for debugging failing tests.

The testing framework executes each property multiple times with different random inputs. Statistical coverage accumulates: even though each individual test uses random data, running enough iterations ensures edge cases appear with high probability. A test run might check empty arrays, single-element arrays, arrays with duplicates, sorted arrays, reverse-sorted arrays, and arrays with equal elements—all without explicitly programming those cases.

Counterexample reporting presents minimal failing cases with the random seed used to generate them. Reports show the original failing input, the shrunk minimal input, and the assertion that failed. Developers can reproduce exact failures by reusing the random seed, maintaining deterministic debugging despite randomized testing.

Ruby Implementation

Ruby property-based testing requires external gems since the standard library lacks built-in support. The ecosystem provides several implementations with different approaches and feature sets.

rspec-rantly integrates property-based testing into RSpec. The gem wraps the Rantly library to provide property definitions within RSpec describe blocks:

require 'rspec'
require 'rspec/rantly'

describe "String#reverse" do
  it "reverses twice equals original" do
    property_of {
      sized(100) { string }
    }.check { |s|
      expect(s.reverse.reverse).to eq(s)
    }
  end
end

The property_of block defines the generator, while check contains the assertion. The sized method controls generation parameters, and string produces random strings. Built-in generators include integer, array, choose, and guard for filtered values.

Generators compose using blocks:

property_of {
  len = range(1, 100)
  array(len) { integer }
}.check { |arr|
  sorted = arr.sort
  expect(sorted.length).to eq(arr.length)
}

propcheck provides a more sophisticated implementation with better shrinking and generator composition:

require 'propcheck'

Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  sorted = array.sort
  sorted.each_cons(2).all? { |a, b| a <= b }
end

The Generators module provides building blocks: integer, string, array, hash, one_of, frequency, and bind for dependent generation. Generators return lazy enumerables that produce infinite sequences of values:

# Generator for non-empty arrays
non_empty_arrays = Generators.array(
  Generators.integer,
  min: 1
)

# Generator for sorted arrays
sorted_arrays = Generators.array(Generators.integer).map(&:sort)

# Generator for user data
user_gen = Generators.tuple(
  name: Generators.string,
  age: Generators.integer(0..120),
  email: Generators.ascii_email
).map { |name:, age:, email:| User.new(name, age, email) }

Custom generators handle domain-specific data structures. A generator for binary trees:

module TreeGenerators
  def self.leaf
    Generators.constant(nil)
  end
  
  def self.node(depth)
    return leaf if depth <= 0
    
    Generators.tuple(
      value: Generators.integer,
      left: Generators.lazy { node(depth - 1) },
      right: Generators.lazy { node(depth - 1) }
    ).map { |value:, left:, right:|
      Node.new(value, left, right)
    }
  end
end

The lazy wrapper prevents infinite recursion by deferring generator evaluation. The depth parameter controls tree size during generation.

Filtering generators excludes invalid values:

# Positive integers only
positive = Generators.integer.where { |n| n > 0 }

# Email addresses with specific domain
company_emails = Generators.ascii_email.where { |e| 
  e.end_with?('@example.com') 
}

Filtering reduces generation efficiency when filters reject most values. Bias generators toward valid values instead:

# Better: generate valid values directly
positive = Generators.integer(1..1000)
company_emails = Generators.tuple(
  name: Generators.ascii_identifier,
).map { |name:| "#{name}@example.com" }

Shrinking configuration controls how aggressively the framework reduces failing cases:

Propcheck.forall(
  array: Generators.array(Generators.integer),
  max_shrink_steps: 1000
) do |array:|
  # property assertion
end

The max_shrink_steps parameter limits shrinking iterations. Higher values find smaller counterexamples at the cost of slower test runs when failures occur.

Integration with test frameworks varies by library. rspec-rantly integrates directly into RSpec. For minitest:

class TestStringOperations < Minitest::Test
  def test_reverse_property
    Propcheck.forall(string: Generators.string) do |string:|
      string.reverse.reverse == string
    end
  end
end

Test failures report both the original failing case and the shrunk minimal example, along with the random seed for reproduction.

Practical Examples

Testing collection operations verifies fundamental properties like element preservation and ordering guarantees:

# Array sorting preserves elements
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  sorted = array.sort
  sorted.sort == sorted && # idempotent
    sorted.length == array.length && # preserves count
    array.all? { |elem| sorted.include?(elem) } && # preserves elements
    sorted.each_cons(2).all? { |a, b| a <= b } # maintains order
end

# Array reverse properties
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  reversed = array.reverse
  reversed.reverse == array && # involution
    reversed.length == array.length && # preserves length
    reversed.first == array.last && # swaps ends
    reversed.zip(array.reverse).all? { |a, b| a == b } # element correspondence
end

These properties encode multiple assertions: idempotence (applying twice equals applying once), element preservation, length invariance, and ordering guarantees. A single property validates behavior across thousands of inputs including edge cases like empty arrays, single elements, duplicates, and large datasets.

Testing string transformations validates encoding conversions, case operations, and formatting functions:

# Case conversion roundtrips
Propcheck.forall(string: Generators.printable_ascii_string) do |string:|
  downcased = string.downcase
  upcased = string.upcase
  
  downcased.upcase.downcase == downcased && # downcase idempotent
    upcased.downcase.upcase == upcased && # upcase idempotent
    downcased.length == string.length && # preserves length
    upcased.length == string.length
end

# String splitting and joining
Propcheck.forall(
  string: Generators.printable_ascii_string,
  delimiter: Generators.ascii_char
) do |string:, delimiter:|
  next true if string.include?(delimiter) # skip ambiguous cases
  
  parts = string.split(delimiter)
  rejoined = parts.join(delimiter)
  rejoined == string
end

The delimiter example shows property preconditions using next true to skip inputs that violate assumptions. When the string contains the delimiter, splitting and rejoining produces different output, so those cases exit early with success.

Testing mathematical operations verifies algebraic properties and numeric relationships:

# Addition properties
Propcheck.forall(
  a: Generators.integer(-1000..1000),
  b: Generators.integer(-1000..1000),
  c: Generators.integer(-1000..1000)
) do |a:, b:, c:|
  (a + b) + c == a + (b + c) && # associativity
    a + b == b + a && # commutativity
    a + 0 == a && # identity
    a + (-a) == 0 # inverse
end

# Multiplication distributes over addition
Propcheck.forall(
  a: Generators.integer(-100..100),
  b: Generators.integer(-100..100),
  c: Generators.integer(-100..100)
) do |a:, b:, c:|
  a * (b + c) == (a * b) + (a * c)
end

Range constraints prevent integer overflow while maintaining coverage. The examples test fundamental mathematical laws that should hold regardless of specific values.

Testing data serialization ensures encoding and decoding preserve information:

# JSON roundtrip
Propcheck.forall(
  data: Generators.json_value
) do |data:|
  parsed = JSON.parse(JSON.generate(data))
  parsed == data
end

# Custom serialization
Propcheck.forall(
  user: user_generator
) do |user:|
  serialized = UserSerializer.serialize(user)
  deserialized = UserSerializer.deserialize(serialized)
  
  deserialized.name == user.name &&
    deserialized.age == user.age &&
    deserialized.email == user.email
end

Serialization properties catch encoding edge cases: special characters, null values, numeric precision loss, nested structures, and circular references.

Testing API boundaries validates input validation and error handling:

# Email validation accepts valid formats
Propcheck.forall(email: Generators.ascii_email) do |email:|
  EmailValidator.valid?(email) == true
end

# Email validation rejects invalid formats
Propcheck.forall(string: Generators.string) do |string:|
  next true if string =~ /\A[^@]+@[^@]+\.[^@]+\z/ # skip actually valid emails
  
  EmailValidator.valid?(string) == false
end

# Password strength requires minimum length
Propcheck.forall(password: Generators.string(0..7)) do |password:|
  PasswordValidator.strong?(password) == false
end

Propcheck.forall(password: Generators.string(8..100)) do |password:|
  # at least not rejected purely on length
  result = PasswordValidator.strong?(password)
  result.is_a?(TrueClass) || result.is_a?(FalseClass)
end

These examples separate positive cases (valid inputs that should pass) from negative cases (invalid inputs that should fail). Generators target boundary conditions around validation rules.

Testing stateful systems verifies state transitions and invariant maintenance:

# Shopping cart maintains count invariants
Propcheck.forall(
  operations: Generators.array(
    Generators.one_of(
      Generators.tuple(op: Generators.constant(:add), item_id: Generators.integer(1..100)),
      Generators.tuple(op: Generators.constant(:remove), item_id: Generators.integer(1..100))
    ),
    max: 50
  )
) do |operations:|
  cart = ShoppingCart.new
  
  operations.each do |operation|
    case operation[:op]
    when :add
      cart.add(operation[:item_id])
    when :remove
      cart.remove(operation[:item_id])
    end
  end
  
  # Invariants that must always hold
  cart.total_items >= 0 && # never negative
    cart.items.all? { |id, count| count > 0 } && # no zero quantities
    cart.total_items == cart.items.values.sum # consistent count
end

This generates random sequences of add/remove operations and verifies invariants hold after each sequence. The approach catches state management bugs that example-based tests miss.

Tools & Ecosystem

rspec-rantly combines RSpec matchers with property-based testing. Installation via Gemfile:

gem 'rspec-rantly'

The library provides generators for primitive types and collection structures. Tests define properties using property_of blocks that integrate with RSpec's reporting and filtering mechanisms. Failures include shrunk counterexamples and random seeds for reproduction.

Feature set: basic generators (integer, string, array, boolean), generator combinators (sized, choose, guard), limited shrinking support, RSpec integration. The library suits teams already using RSpec who want to add property-based tests without learning new syntax.

Limitations include less sophisticated shrinking compared to other implementations and limited generator composition capabilities for complex data structures.

propcheck implements advanced property-based testing with extensive shrinking and generator composition. Installation:

gem 'propcheck'

The library models generators as lazy enumerables supporting map, filter, and bind operations. Shrinking uses a tree-based search strategy that finds minimal counterexamples efficiently.

Feature set: comprehensive generator library, sophisticated shrinking, generator composition via functors and monads, configurable test runs, custom generator definition. The library provides the most complete implementation of QuickCheck-style testing in Ruby.

# Advanced generator composition
user_with_posts = Generators.tuple(
  user: user_generator,
  posts: Generators.array(post_generator, min: 0, max: 10)
).where { |user:, posts:| posts.all? { |p| p.author_id == user.id } }

Propcheck handles complex data dependencies and maintains shrinking effectiveness across composed generators.

test-prof integration allows performance profiling of property-based tests. Since property tests execute many iterations, identifying slow properties improves test suite performance:

require 'test_prof'

# Profile property test performance
TestProf::EventProf.monitor(Propcheck, 'propcheck.example') do
  Propcheck.forall(data: expensive_generator) do |data:|
    # test assertions
  end
end

The profiling reveals which generators or assertions consume the most time, guiding optimization efforts.

custom test runners execute property tests in continuous integration with configurable iteration counts:

# config/propcheck.rb
Propcheck.configure do |config|
  config.n_runs = ENV['CI'] ? 1000 : 100
  config.max_shrink_steps = 500
  config.verbose = true if ENV['VERBOSE']
end

Production CI runs more iterations than local development, balancing test coverage against execution time.

integration with mutation testing combines property-based tests with mutant or similar tools. Property tests provide strong specifications that detect a high percentage of code mutations:

# Property-based tests catch more mutations
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  sorted = CustomSort.sort(array)
  sorted.each_cons(2).all? { |a, b| a <= b }
end

A mutation changing < to <= in the sorting implementation fails the property test, while example-based tests might miss the mutation if they lack duplicate values.

Common Patterns

Roundtrip properties verify encoding and decoding preserve data:

# Serialization roundtrip
Propcheck.forall(object: object_generator) do |object:|
  encoded = encode(object)
  decoded = decode(encoded)
  decoded == object
end

# Coordinate transformation roundtrip
Propcheck.forall(
  x: Generators.float(-1000.0..1000.0),
  y: Generators.float(-1000.0..1000.0)
) do |x:, y:|
  polar = cartesian_to_polar(x, y)
  back = polar_to_cartesian(*polar)
  
  (back[0] - x).abs < 0.001 && # floating point tolerance
    (back[1] - y).abs < 0.001
end

Roundtrip patterns catch information loss during transformations. The coordinate example shows tolerance for floating-point arithmetic errors.

Idempotence properties assert that applying an operation multiple times equals applying it once:

# Sorting is idempotent
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  sort(sort(array)) == sort(array)
end

# String normalization is idempotent
Propcheck.forall(string: Generators.string) do |string:|
  normalized = normalize(string)
  normalize(normalized) == normalized
end

# Set operations are idempotent
Propcheck.forall(set: Generators.array(Generators.integer)) do |set:|
  unique = set.uniq
  unique.uniq == unique
end

Idempotence properties ensure operations reach a stable state regardless of repeated application.

Invariant properties verify characteristics that remain constant across operations:

# Filtering preserves order
Propcheck.forall(
  array: Generators.array(Generators.integer)
) do |array:|
  filtered = array.select { |n| n > 0 }
  
  # Original relative order maintained
  filtered == array.select { |n| n > 0 }
end

# Map preserves length
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  mapped = array.map { |n| n * 2 }
  mapped.length == array.length
end

Invariant patterns document expectations about operation behavior that should hold universally.

Commutativity properties verify operation order independence:

# Set union is commutative
Propcheck.forall(
  a: Generators.array(Generators.integer),
  b: Generators.array(Generators.integer)
) do |a:, b:|
  union_ab = (a + b).uniq.sort
  union_ba = (b + a).uniq.sort
  union_ab == union_ba
end

# Hash merge order matters (not commutative)
Propcheck.forall(
  h1: Generators.hash(key: Generators.string, value: Generators.integer),
  h2: Generators.hash(key: Generators.string, value: Generators.integer)
) do |h1:, h2:|
  # This property should fail for shared keys
  h1.merge(h2) == h2.merge(h1)
end

The hash merge example demonstrates testing negative properties—operations that should not commute. The property fails, confirming merge behavior depends on argument order.

Associativity properties verify grouping independence:

# String concatenation is associative
Propcheck.forall(
  a: Generators.string,
  b: Generators.string,
  c: Generators.string
) do |a:, b:, c:|
  (a + b) + c == a + (b + c)
end

# Array concatenation is associative
Propcheck.forall(
  a: Generators.array(Generators.integer),
  b: Generators.array(Generators.integer),
  c: Generators.array(Generators.integer)
) do |a:, b:, c:|
  (a + b) + c == a + (b + c)
end

Associativity enables optimization through operation reordering without affecting results.

Oracle properties compare implementation against trusted reference:

# Compare custom implementation to standard library
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  custom_sorted = CustomSort.sort(array)
  reference_sorted = array.sort
  custom_sorted == reference_sorted
end

# Compare optimized version to simple version
Propcheck.forall(n: Generators.integer(0..1000)) do |n:|
  optimized_result = optimized_fibonacci(n)
  simple_result = simple_fibonacci(n)
  optimized_result == simple_result
end

Oracle patterns verify new implementations against known-correct versions, catching logic errors while maintaining performance improvements.

Metamorphic properties relate multiple function executions:

# Reversing twice returns original
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  array.reverse.reverse == array
end

# Negating twice returns original
Propcheck.forall(n: Generators.integer) do |n:|
  -(-n) == n
end

# Multiple sort keys produce refinement
Propcheck.forall(
  users: Generators.array(user_generator)
) do |users:|
  by_age = users.sort_by(&:age)
  by_age_then_name = users.sort_by { |u| [u.age, u.name] }
  
  # Age ordering preserved when adding name
  ages_match = by_age.map(&:age) == by_age_then_name.map(&:age)
  ages_match
end

Metamorphic patterns express relationships between different function invocations, catching subtle behavior differences.

Common Pitfalls

Over-constrained generators produce insufficient variety. Filtering or narrow ranges limit test coverage:

# Too constrained - only tests small positive numbers
Propcheck.forall(n: Generators.integer(1..10)) do |n:|
  factorial(n) > 0
end

# Better - tests full range including edge cases
Propcheck.forall(n: Generators.integer(0..100)) do |n:|
  result = factorial(n)
  n == 0 ? result == 1 : result > 0
end

Generators should cover the full input domain including boundaries, negative values, empty collections, and extreme sizes. Overly restrictive generators miss edge cases that cause failures.

Flaky properties pass or fail inconsistently due to random variation:

# Flaky - depends on random distribution
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  sorted = array.sort
  # This might fail if array happens to be pre-sorted
  sorted != array
end

# Fixed - property that always holds
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  sorted = array.sort
  sorted.each_cons(2).all? { |a, b| a <= b }
end

Properties must hold for all valid inputs, not most inputs. Statistical properties that hold "usually" cause intermittent failures that obscure real bugs.

Hidden preconditions cause false failures when properties assume input constraints:

# Assumes non-empty array but doesn't enforce it
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  max = array.max # fails for empty array
  array.all? { |n| n <= max }
end

# Fixed - handle empty case or constrain generator
Propcheck.forall(array: Generators.array(Generators.integer, min: 1)) do |array:|
  max = array.max
  array.all? { |n| n <= max }
end

Explicitly model preconditions in generators rather than assuming them in properties. Alternatively, handle edge cases within properties using guards.

Inefficient shrinking produces large counterexamples that obscure root causes:

# Poor shrinking - custom types without shrink support
class CustomData
  attr_reader :values
  
  def initialize(values)
    @values = values
  end
end

def custom_generator
  Generators.array(Generators.integer).map { |vals| CustomData.new(vals) }
end

The generator produces CustomData instances, but the framework cannot shrink them effectively. Failures report large CustomData objects rather than minimal examples.

Fix by implementing shrinking for custom types or using primitive types in generators:

# Better - work with primitives
Propcheck.forall(values: Generators.array(Generators.integer)) do |values:|
  data = CustomData.new(values)
  # test using data
end

Expensive generators slow test execution:

# Slow - generates complex nested structures
def slow_generator
  Generators.array(
    Generators.array(
      Generators.hash(
        key: Generators.string(10..100),
        value: Generators.array(Generators.integer, max: 100)
      ),
      max: 50
    ),
    max: 50
  )
end

Deeply nested generators with large sizes create performance problems. Balance coverage against execution time by tuning size parameters and reducing nesting depth.

Weak properties pass for incorrect implementations:

# Too weak - doesn't actually verify sorting
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  sorted = buggy_sort(array)
  sorted.length == array.length # only checks length preservation
end

# Strong property catches bugs
Propcheck.forall(array: Generators.array(Generators.integer)) do |array:|
  sorted = buggy_sort(array)
  sorted.each_cons(2).all? { |a, b| a <= b } && # verifies ordering
    sorted.length == array.length && # verifies length
    array.all? { |elem| sorted.include?(elem) } # verifies elements
end

Properties should fully specify expected behavior. Partial specifications let bugs slip through.

Ignoring shrinking output wastes debugging effort. When tests fail, examine the shrunk counterexample first:

Property failed after 47 test cases
Original failing input: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Shrunk to: [1, 2]

The shrunk input [1, 2] triggers the same failure with minimal complexity, making the bug easier to locate than debugging with the original 10-element array.

Determinism issues prevent failure reproduction. Property tests use random seeds, but some test environments reset seeds between runs:

# Fails, but cannot reproduce because seed changes
Propcheck.forall(data: generator) do |data:|
  flaky_operation(data)
end

Configure fixed seeds for debugging:

Propcheck.forall(
  data: generator,
  seed: 12345 # from failure report
) do |data:|
  flaky_operation(data)
end

Always capture and reuse seeds from failure reports to reproduce bugs deterministically.

Reference

Property Pattern Catalog

Generator Types

Shrinking Strategies

Configuration Options

Common Property Assertions

Test Execution Parameters