Crash Dump Analysis

Overview

Crash dump analysis examines the state of a program at the moment of failure by inspecting memory snapshots, stack traces, and system state information. When a program terminates unexpectedly, the operating system can capture a snapshot of the process memory, register states, and execution context. This snapshot, called a crash dump or core dump, preserves the exact state that led to the failure.

The practice originated in mainframe computing where operators would print memory contents to paper for analysis. Modern crash dumps contain thread stacks, heap allocations, loaded libraries, register values, and file descriptors. Analysis reconstructs the sequence of events leading to failure without requiring the ability to reproduce the crash.

Crash dumps serve multiple purposes in software development. They enable debugging of issues that occur in production environments where debuggers cannot run. They capture failures that occur intermittently or under specific conditions that are difficult to reproduce. They provide forensic evidence for security incidents and help identify memory corruption, race conditions, and resource exhaustion problems.

The analysis process differs significantly from interactive debugging. Interactive debuggers allow stepping through code execution and examining state dynamically. Crash dump analysis works with a static snapshot, requiring different techniques to understand program flow and identify root causes. Analysts must infer execution history from memory contents and stack traces without the ability to execute code or evaluate expressions in the original context.

# Basic crash dump generation in Ruby
Signal.trap("SEGV") do
  File.write("/tmp/crash_info.txt", "Segfault occurred at #{Time.now}")
  Thread.list.each do |thread|
    File.open("/tmp/crash_info.txt", "a") do |f|
      f.puts "\nThread: #{thread.inspect}"
      f.puts thread.backtrace.join("\n") if thread.backtrace
    end
  end
  exit(1)
end

Key Principles

Crash dumps contain multiple types of information critical for failure analysis. The stack trace shows the call chain at the time of crash, revealing which functions were executing and how they were invoked. Each stack frame contains local variables, function parameters, and the instruction pointer showing the exact location in code where execution stopped. The heap section contains dynamically allocated objects with their current values, revealing memory corruption or invalid references.

Memory layout information shows how the process address space was organized at crash time. This includes the locations of code segments, data segments, heap regions, and thread stacks. Understanding memory layout helps identify buffer overflows, use-after-free errors, and heap corruption. The dump also contains register values, which are critical for understanding the precise CPU state during the crash.

Thread state information captures what each thread was doing at crash time. This includes thread IDs, priority levels, synchronization state, and per-thread stacks. Multi-threaded crashes often involve race conditions or deadlocks, making thread state analysis essential. The dump shows which threads were running, blocked, or waiting on synchronization primitives.

Signal and exception information records what caused the crash. On Unix systems, signals like SIGSEGV indicate memory access violations, while SIGABRT indicates deliberate termination. On Windows, structured exceptions provide similar information. The signal context includes the faulting address for memory violations and the exception code for other failures.

File descriptors and system resources show what external resources the program was using. Open files, network connections, and inter-process communication channels all appear in the dump. This information helps identify resource leaks or improper resource handling that contributed to the crash.

The concept of dump completeness varies by platform and configuration. A minidump contains minimal information like stack traces and basic thread state, suitable for quick analysis or privacy-sensitive environments. A full dump captures the entire process address space, enabling deep analysis but consuming significant storage. Heap dumps focus specifically on memory allocations, used primarily for memory leak analysis.

# Capturing comprehensive crash context
class CrashHandler
  def self.install
    Signal.trap("SEGV") { handle_crash("SIGSEGV") }
    Signal.trap("ABRT") { handle_crash("SIGABRT") }
    Signal.trap("ILL") { handle_crash("SIGILL") }
  end

  def self.handle_crash(signal)
    context = {
      signal: signal,
      timestamp: Time.now,
      pid: Process.pid,
      threads: capture_threads,
      memory: capture_memory_info,
      environment: ENV.to_h
    }
    
    File.write("/var/crash/dump_#{Process.pid}.json", JSON.pretty_generate(context))
    abort
  end

  def self.capture_threads
    Thread.list.map do |t|
      {
        id: t.object_id,
        status: t.status,
        backtrace: t.backtrace || []
      }
    end
  end

  def self.capture_memory_info
    if RUBY_PLATFORM =~ /linux/
      File.read("/proc/#{Process.pid}/status")
    else
      "Memory info not available on this platform"
    end
  end
end

Ruby Implementation

Ruby's crash handling differs from lower-level languages because Ruby runs in a virtual machine. When a Ruby process crashes, the dump might reflect issues in either the Ruby interpreter itself or in C extensions. Ruby provides signal handling mechanisms through the Signal module, allowing programs to intercept signals before they terminate the process.

The built-in signal handlers can capture information about the Ruby execution state at crash time. This includes the current exception if one was being raised, the backtrace of all threads, and Ruby's internal object state. However, Ruby signal handlers run within the Ruby VM, so they cannot capture information if the VM itself is corrupted.

# Ruby-specific crash dump with VM state
require 'json'

module RubyCrashDumper
  def self.install
    %w[SEGV ABRT ILL BUS FPE].each do |sig|
      Signal.trap(sig) { dump_ruby_state(sig) }
    end
  end

  def self.dump_ruby_state(signal)
    dump_data = {
      signal: signal,
      timestamp: Time.now.iso8601,
      ruby_version: RUBY_VERSION,
      platform: RUBY_PLATFORM,
      exception: capture_current_exception,
      threads: capture_all_threads,
      objects: capture_object_counts,
      gc_stats: GC.stat,
      loaded_features: $LOADED_FEATURES.size
    }

    dump_file = "/tmp/ruby_crash_#{Process.pid}_#{Time.now.to_i}.json"
    File.write(dump_file, JSON.pretty_generate(dump_data))
    
    STDERR.puts "Crash dump written to #{dump_file}"
    exit!(1)
  end

  def self.capture_current_exception
    return nil unless $!
    {
      class: $!.class.name,
      message: $!.message,
      backtrace: $!.backtrace
    }
  end

  def self.capture_all_threads
    Thread.list.map do |thread|
      {
        object_id: thread.object_id,
        status: thread.status,
        priority: thread.priority,
        backtrace: thread.backtrace,
        variables: capture_thread_locals(thread)
      }
    end
  end

  def self.capture_thread_locals(thread)
    locals = {}
    thread.thread_variables.each do |var|
      locals[var] = thread.thread_variable_get(var).inspect rescue "Error capturing"
    end
    locals
  end

  def self.capture_object_counts
    counts = Hash.new(0)
    ObjectSpace.each_object { |obj| counts[obj.class.name] += 1 }
    counts.sort_by { |_, count| -count }.take(20).to_h
  end
end

RubyCrashDumper.install

Ruby on Unix systems can trigger core dumps through the operating system. Setting Process.setrlimit(Process::RLIMIT_CORE, Process::RLIM_INFINITY) enables core dump generation. The resulting core file can be analyzed with debuggers like gdb or lldb, though interpreting Ruby VM internals requires understanding of Ruby's C implementation.

C extensions in Ruby applications introduce additional complexity. When a C extension crashes, the Ruby VM may not have an opportunity to run crash handlers. The operating system generates a core dump directly, capturing both the Ruby VM state and the C extension's state. Analyzing these dumps requires tools that understand both Ruby internals and C memory layout.

# Enabling system core dumps
Process.setrlimit(Process::RLIMIT_CORE, Process::RLIM_INFINITY)

# Verify core dump settings
core_limit = Process.getrlimit(Process::RLIMIT_CORE)
puts "Core dump size limit: #{core_limit[0]} bytes"

# Core pattern can be checked on Linux
if File.exist?("/proc/sys/kernel/core_pattern")
  pattern = File.read("/proc/sys/kernel/core_pattern").strip
  puts "Core dump pattern: #{pattern}"
end

The Ruby garbage collector's state provides valuable debugging information. When analyzing crashes related to memory issues, GC statistics reveal memory pressure, allocation rates, and collection frequency. This information helps distinguish between crashes caused by memory exhaustion versus other issues.

# Comprehensive GC state capture
class GCDumper
  def self.capture_state
    {
      stat: GC.stat,
      latest_gc_info: GC.latest_gc_info,
      stress: GC.stress,
      count: GC.count,
      time: GC.total_time,
      profiler: capture_gc_profiler
    }
  end

  def self.capture_gc_profiler
    GC::Profiler.enable
    report = GC::Profiler.result
    GC::Profiler.clear
    report
  end
end

Ruby's Exception class provides programmatic access to backtraces, which form the foundation of crash analysis. The backtrace_locations method returns structured information about each stack frame, including file paths, line numbers, and method names. This structured data allows automated analysis of crash patterns.

Practical Examples

Analyzing a segmentation fault in a Ruby application requires examining both Ruby-level and system-level information. The process begins by identifying the crashing thread and its stack trace, then examining the state of objects being accessed when the crash occurred.

# Scenario: Analyzing a crash from a C extension
require 'json'

class CrashAnalyzer
  def analyze_dump(dump_file)
    data = JSON.parse(File.read(dump_file))
    
    puts "=== Crash Analysis Report ==="
    puts "Signal: #{data['signal']}"
    puts "Time: #{data['timestamp']}"
    puts "Ruby: #{data['ruby_version']} on #{data['platform']}"
    puts
    
    identify_crashing_thread(data['threads'])
    analyze_object_state(data['objects'])
    check_memory_pressure(data['gc_stats'])
    
    if data['exception']
      puts "\n=== Active Exception ==="
      puts "#{data['exception']['class']}: #{data['exception']['message']}"
      puts data['exception']['backtrace'].first(10).join("\n")
    end
  end

  def identify_crashing_thread(threads)
    puts "=== Thread Analysis ==="
    puts "Active threads: #{threads.size}"
    
    threads.each_with_index do |thread, idx|
      next unless thread['backtrace']
      
      puts "\nThread #{idx} (#{thread['status']}):"
      suspicious = detect_suspicious_frames(thread['backtrace'])
      if suspicious.any?
        puts "  SUSPICIOUS FRAMES DETECTED:"
        suspicious.each { |frame| puts "    #{frame}" }
      else
        puts "  Top frame: #{thread['backtrace'].first}"
      end
    end
  end

  def detect_suspicious_frames(backtrace)
    suspicious = []
    backtrace.each do |frame|
      suspicious << frame if frame.include?('.so') # C extension
      suspicious << frame if frame.include?('native') # Native code
      suspicious << frame if frame.include?('malloc') || frame.include?('free')
    end
    suspicious
  end

  def analyze_object_state(objects)
    puts "\n=== Object Analysis ==="
    puts "Top object types:"
    objects.take(10).each do |type, count|
      puts "  #{type}: #{count}"
    end
    
    # Check for memory leak indicators
    if objects['String'] && objects['String'] > 1_000_000
      puts "  WARNING: Excessive String objects may indicate memory leak"
    end
    
    if objects['Array'] && objects['Array'] > 500_000
      puts "  WARNING: Excessive Array objects may indicate memory leak"
    end
  end

  def check_memory_pressure(gc_stats)
    puts "\n=== Memory Pressure Analysis ==="
    puts "GC count: #{gc_stats['count']}"
    puts "Heap allocated pages: #{gc_stats['heap_allocated_pages']}"
    puts "Heap free slots: #{gc_stats['heap_free_slots']}"
    puts "Old objects: #{gc_stats['old_objects']}"
    
    if gc_stats['heap_free_slots'] < 1000
      puts "  WARNING: Very low free slots, memory exhaustion likely"
    end
    
    if gc_stats['count'] > 10000
      puts "  WARNING: High GC count may indicate memory pressure"
    end
  end
end

# Usage
analyzer = CrashAnalyzer.new
analyzer.analyze_dump("/tmp/ruby_crash_12345_1640000000.json")

A deadlock scenario demonstrates the importance of thread state analysis. When multiple threads block waiting for each other, the crash dump shows threads in waiting states, often with synchronization primitives visible in stack traces.

# Scenario: Detecting deadlock from crash dump
class DeadlockDetector
  def analyze_for_deadlock(dump_file)
    data = JSON.parse(File.read(dump_file))
    threads = data['threads']
    
    waiting_threads = threads.select { |t| t['status'] == 'sleep' }
    
    if waiting_threads.size == threads.size && threads.size > 1
      puts "DEADLOCK SUSPECTED: All threads are waiting"
      
      # Analyze what each thread is waiting on
      waiting_threads.each_with_index do |thread, idx|
        puts "\nThread #{idx}:"
        mutex_frames = thread['backtrace'].select { |f| f.include?('Mutex') || f.include?('lock') }
        mutex_frames.each { |frame| puts "  Waiting: #{frame}" }
      end
      
      puts "\n=== Deadlock Resolution Suggestions ==="
      puts "1. Check for circular lock dependencies"
      puts "2. Review lock acquisition order"
      puts "3. Consider using timeout on lock acquisition"
    else
      puts "No deadlock pattern detected"
    end
  end
end

Memory corruption issues often manifest as crashes in seemingly unrelated code. The corruption occurs at one point, but the program doesn't crash until it accesses the corrupted memory later. Crash dump analysis must work backwards from the crash site to find the corruption source.

# Scenario: Tracking memory corruption patterns
class CorruptionAnalyzer
  def analyze_heap_corruption(dump_file)
    data = JSON.parse(File.read(dump_file))
    
    # Look for corruption indicators
    indicators = []
    
    # Check for double-free patterns
    data['threads'].each do |thread|
      thread['backtrace']&.each do |frame|
        if frame.include?('free') || frame.include?('gc_')
          indicators << {
            type: 'potential_double_free',
            thread: thread['object_id'],
            frame: frame
          }
        end
      end
    end
    
    # Check for buffer overflow patterns
    if data['signal'] == 'SIGSEGV'
      indicators << {
        type: 'memory_access_violation',
        details: 'Segmentation fault suggests invalid memory access'
      }
    end
    
    # Analyze object counts for anomalies
    if data['objects']
      data['objects'].each do |type, count|
        if count > 10_000_000
          indicators << {
            type: 'excessive_allocation',
            class: type,
            count: count
          }
        end
      end
    end
    
    report_findings(indicators)
  end

  def report_findings(indicators)
    puts "=== Memory Corruption Analysis ==="
    if indicators.empty?
      puts "No clear corruption indicators found"
    else
      indicators.each do |indicator|
        puts "\n#{indicator[:type].upcase}:"
        indicator.each { |k, v| puts "  #{k}: #{v}" unless k == :type }
      end
    end
  end
end

Tools & Ecosystem

Ruby developers use multiple tools for crash dump analysis, each serving different purposes. The standard Ruby interpreter provides built-in crash reporting through signal handlers and exception mechanisms. When the Ruby VM crashes due to internal errors, it writes a detailed error report showing the C-level backtrace, Ruby-level backtrace, and system information.

Third-party gems extend crash analysis capabilities. The rbtrace gem attaches to running Ruby processes to inspect their state without stopping execution. While not a crash dump analyzer per se, it helps understand process state before crashes occur. The heapy gem analyzes heap dumps generated by Ruby's heap dump facility, identifying memory leaks and retention issues.

# Using Ruby's built-in heap dump
require 'objspace'

# Generate heap dump before potential crash
ObjectSpace.trace_object_allocations_start

# Simulate some work
1000.times { Array.new(1000) { "data" } }

# Dump heap to file
File.open('/tmp/heap_dump.json', 'w') do |f|
  ObjectSpace.dump_all(output: f)
end

ObjectSpace.trace_object_allocations_stop

System-level debugging tools analyze core dumps generated by the operating system. On Linux, gdb (GNU Debugger) opens core files and inspects the crashed process's memory. For Ruby applications, gdb shows the C-level stack trace of the Ruby VM, which is less useful without Ruby-specific extensions.

# Analyzing a Ruby core dump with gdb
# gdb /usr/bin/ruby core.12345
# (gdb) bt           # C-level backtrace
# (gdb) info threads # Show all threads
# (gdb) thread apply all bt # Backtrace for all threads

The ruby-gdb.py helper script, included with Ruby source, adds Ruby-aware commands to gdb. These commands display Ruby stack traces, inspect Ruby objects, and navigate Ruby's internal structures. Installing these helpers transforms gdb from a C debugger into a Ruby-aware debugging tool.

Platform-specific tools provide additional capabilities. On macOS, lldb serves as the primary debugger with similar capabilities to gdb. The Console application displays crash reports automatically generated by the system. On Windows, WinDbg analyzes crash dumps with extensions that understand Ruby VM internals.

Profiling tools help prevent crashes by identifying issues before they cause failures. The stackprof gem performs sampling-based profiling to find hot spots and deep recursion. The memory_profiler gem tracks object allocations to identify memory leaks. The derailed_benchmarks gem specializes in Rails application memory analysis.

# Using memory_profiler to prevent future crashes
require 'memory_profiler'

report = MemoryProfiler.report do
  # Code that might be leaking memory
  @cache = {}
  10_000.times do |i|
    @cache[i] = "data" * 1000
  end
end

report.pretty_print

# Check for retained memory
if report.total_retained_memsize > 100_000_000
  puts "WARNING: High retained memory detected"
  File.write('/tmp/memory_report.txt', report.pretty_print)
end

Application Performance Monitoring (APM) services like New Relic, Datadog, and Scout provide crash tracking and analysis in production. These services capture exceptions, collect performance metrics, and aggregate crash reports across deployments. They offer dashboards showing crash frequency, affected users, and trends over time.

Crash reporting services specialize in collecting and analyzing crashes from production environments. Services like Sentry, Bugsnag, and Rollbar capture exceptions with full context including environment variables, request parameters, and user information. They group similar crashes together and track resolution status.

# Integrating with Sentry for crash tracking
require 'sentry-ruby'

Sentry.init do |config|
  config.dsn = ENV['SENTRY_DSN']
  config.breadcrumbs_logger = [:active_support_logger, :http_logger]
  config.traces_sample_rate = 0.1
  
  # Add custom context to crashes
  config.before_send = lambda do |event, hint|
    event.contexts[:runtime] = {
      name: 'ruby',
      version: RUBY_VERSION,
      gc_stats: GC.stat
    }
    event
  end
end

# Crashes are automatically reported
begin
  risky_operation
rescue => e
  Sentry.capture_exception(e)
  raise
end

Error Handling & Edge Cases

Signal handling during crash dump generation introduces complexity. Signal handlers run in an async-signal-safe context with significant restrictions. They cannot allocate memory, cannot call non-reentrant functions, and must avoid most Ruby operations. Violating these restrictions causes undefined behavior, potentially preventing crash dump generation entirely.

# Signal handler with safety considerations
Signal.trap("SEGV") do
  # UNSAFE: These operations might fail in signal context
  # File.write - might allocate memory
  # JSON.generate - definitely allocates memory
  # Complex Ruby operations - might trigger GC
  
  # SAFER: Write minimal information with preallocated resources
  fd = IO.sysopen('/tmp/crash.log', 'a')
  IO.write(fd, "SEGV at #{Time.now.to_i}\n")
  IO.close(fd)
  
  exit!(1) # exit! is async-signal-safe, exit is not
end

Recursive crash handling occurs when the crash dump code itself crashes. This happens when examining corrupted memory structures or when signal handlers trigger additional signals. Proper crash handlers include recursion detection to prevent infinite loops.

# Crash handler with recursion protection
module SafeCrashHandler
  @in_handler = false
  
  def self.handle_crash(signal)
    if @in_handler
      # Already handling a crash, exit immediately
      exit!(1)
    end
    
    @in_handler = true
    
    begin
      write_crash_dump(signal)
    rescue => e
      # Even crash handling failed, write minimal info
      STDERR.puts "Crash handler failed: #{e.class}"
    ensure
      exit!(1)
    end
  end
  
  def self.write_crash_dump(signal)
    # Crash dump logic here
  end
end

Insufficient disk space prevents writing crash dumps. Crash handlers should check available space before writing large dumps and fall back to minimal dumps when space is limited. Preallocating space for crash dumps ensures that space is available when needed.

# Checking available disk space before dump
require 'sys/filesystem'

def write_crash_dump_safe(data, path)
  stat = Sys::Filesystem.stat(File.dirname(path))
  available_mb = stat.bytes_available / 1024 / 1024
  
  if available_mb < 100
    # Write minimal dump
    File.write("#{path}.minimal", data[:summary].to_json)
  else
    # Write full dump
    File.write(path, data.to_json)
  end
rescue => e
  # Fall back to STDERR if file writing fails
  STDERR.puts "Crash dump write failed: #{e}"
  STDERR.puts data[:summary].inspect
end

Permission issues prevent writing dumps to protected directories. Production systems often run with restricted permissions, making common dump locations like /var/crash inaccessible. Crash handlers need fallback locations like /tmp or the application's working directory.

Thread-safety issues arise when multiple threads crash simultaneously or when crash handlers access shared state. Multiple threads might trigger signal handlers concurrently, causing race conditions in dump generation. Using thread-local storage and atomic operations prevents these races.

# Thread-safe crash dump counter
require 'concurrent'

module CrashDumper
  @dump_counter = Concurrent::AtomicFixnum.new(0)
  
  def self.generate_dump
    # Each thread gets a unique dump number
    dump_num = @dump_counter.increment
    dump_file = "/tmp/crash_#{Process.pid}_#{dump_num}.json"
    
    # Continue with dump generation
  end
end

Large process crashes generate enormous dumps. A Ruby process with 10GB of heap memory produces a 10GB+ core dump. These dumps are impractical to transfer, store, or analyze. Configuring the kernel to truncate dumps or using minidumps addresses this issue.

# Generating a minidump instead of full dump
class MinidumpGenerator
  def self.generate
    {
      timestamp: Time.now.iso8601,
      pid: Process.pid,
      # Only capture summary information
      thread_count: Thread.list.size,
      top_backtrace: Thread.main.backtrace&.first(20),
      memory_summary: {
        rss_mb: get_rss_mb,
        objects: ObjectSpace.count_objects.slice(:TOTAL, :FREE, :T_STRING, :T_ARRAY, :T_HASH)
      }
    }
  end
  
  def self.get_rss_mb
    return nil unless File.exist?("/proc/#{Process.pid}/status")
    
    status = File.read("/proc/#{Process.pid}/status")
    if match = status.match(/VmRSS:\s+(\d+)\s+kB/)
      match[1].to_i / 1024
    end
  end
end

Common Pitfalls

Assuming crash dumps contain current code leads to misanalysis. The code in memory might differ from the current source code if the process has been running since before a deployment. Always verify the code version present in the dump matches expectations.

Ignoring load time versus crash time causes confusion. Stack traces show where code was executing at crash time, not necessarily where the bug was introduced. A null pointer exception might occur in generic container code, while the actual bug is where null was inserted earlier.

Focusing solely on the crashing thread misses the root cause. The crashing thread might be a victim of corruption caused by another thread. Multi-threaded crashes require analyzing all threads to understand the full picture. A crash in thread A might result from memory corruption by thread B.

Over-interpreting optimized code stack traces causes errors. Optimized builds inline functions, eliminate stack frames, and reorder operations. Stack traces from optimized code show incomplete or confusing call chains. Production builds often enable optimizations, making their crash dumps harder to analyze than debug builds.

Treating core dumps as small data files ignores resource constraints. A core dump of a large Ruby process might be tens of gigabytes. Transferring, storing, and analyzing these dumps requires appropriate infrastructure. Network bandwidth, disk space, and analysis tool memory limits all become relevant.

Modifying the process during dump generation corrupts the dump. Some debugging approaches attach to the crashed process and modify its memory. These modifications invalidate the crash state, making the dump useless for understanding the original failure.

# Common mistake: Complex operations in signal handler
Signal.trap("SEGV") do
  # WRONG: These operations are not async-signal-safe
  logger = Logger.new('/tmp/crash.log')
  logger.error("Crashed: #{$!.inspect}") # Might trigger GC
  
  # WRONG: Network operations in signal handler
  HTTP.post('https://api.example.com/crashes', {
    error: $!.message,
    backtrace: caller
  })
  
  exit(1)
end

# BETTER: Minimal operations only
Signal.trap("SEGV") do
  STDERR.puts "CRASH"
  exit!(1) # Use exit! not exit
end

Analyzing crashes without symbol information makes stack traces cryptic. Stack traces show memory addresses instead of function names without symbols. Production deployments should preserve symbol files or debug symbols to make crash analysis feasible.

Expecting complete information from minidumps leads to frustration. Minidumps sacrifice completeness for size. They capture stack traces but not heap contents, making certain types of analysis impossible. Choose dump type based on analysis needs versus resource constraints.

Dumping sensitive data to crashes creates security issues. Crash dumps contain process memory, including passwords, API keys, and user data. Storing these dumps insecurely or sending them to external services exposes sensitive information. Scrub sensitive data or encrypt dumps before storage.

Reference

Crash Dump Types

Signal Types (Unix)

Ruby Crash Dump Methods

GC Stat Keys

Crash Analysis Commands

Ruby Signal Handler Safety

Common Crash Patterns

Analysis Checklist