Time-Series Data Storage

Overview

Block storage represents data as fixed-size chunks called blocks, each with a unique address. Storage systems present these blocks as raw volumes to operating systems and applications without imposing file system structure. The operating system or application controls how data maps to blocks and manages the file system layer if needed.

Cloud providers offer block storage as persistent volumes that attach to virtual machines, while physical infrastructure implements block storage through storage area networks (SAN) and direct-attached storage (DAS). Applications interact with block storage through standard device interfaces, reading and writing data at the block level.

Block storage differs fundamentally from file storage and object storage. File storage presents a hierarchical directory structure with file metadata, while object storage organizes data as objects with extensive metadata in a flat namespace. Block storage provides raw block access without inherent file organization, offering lower latency and more direct control over storage operations.

# Block storage presents as block devices in the system
# /dev/sda, /dev/xvdf, etc.

# Applications read/write at block level
File.open('/dev/xvdf', 'rb') do |device|
  # Read first 4096 bytes (one block at typical block size)
  block = device.read(4096)
end

Operating systems create file systems on top of block devices, translating file operations into block-level reads and writes. Database systems often bypass file systems entirely, writing directly to block devices for better control over I/O patterns and caching.

Key Principles

Block storage operates on several core principles that define its behavior and characteristics. The storage system divides the total storage capacity into uniform blocks, typically 512 bytes to 64 KB in size. Each block receives a unique logical block address (LBA) that the system uses for addressing operations.

Block Addressing and Access

The storage controller translates logical block addresses to physical locations on storage media. This abstraction allows the system to remap blocks transparently, enabling features like snapshots, replication, and thin provisioning. Applications specify which block to access and whether to read or write, with the storage system handling physical media operations.

# Conceptual block access pattern
class BlockDevice
  BLOCK_SIZE = 4096
  
  def read_block(lba)
    # Calculate offset from logical block address
    offset = lba * BLOCK_SIZE
    # Read one block from physical media
    read_from_media(offset, BLOCK_SIZE)
  end
  
  def write_block(lba, data)
    offset = lba * BLOCK_SIZE
    write_to_media(offset, data)
  end
end

Volume Management

Block storage systems present storage as volumes - logical units that combine multiple physical blocks into a single addressable space. Volumes attach to compute instances as block devices, appearing to the operating system as local disks. The system handles volume provisioning, attachment, detachment, and deletion operations.

Volume snapshots capture point-in-time copies of block data. The storage system implements snapshots through copy-on-write mechanisms, storing only changed blocks to minimize storage consumption and snapshot creation time. Snapshots enable backup, disaster recovery, and volume cloning workflows.

I/O Operations and Consistency

Block storage guarantees atomic writes at the block level. When an application writes a block, either the entire block writes successfully or the operation fails without partial writes. This atomicity forms the foundation for database consistency and file system integrity.

The storage system processes I/O operations sequentially or randomly depending on access patterns. Sequential access reads or writes contiguous blocks in order, while random access jumps between non-contiguous blocks. Performance characteristics differ significantly between these patterns.

Data Durability and Redundancy

Block storage systems implement redundancy to protect against hardware failures. Cloud providers replicate blocks across multiple physical devices within a datacenter or across availability zones. RAID configurations combine multiple physical disks to provide redundancy and performance improvements.

# Volume attachment model
class Volume
  attr_reader :volume_id, :size_gb, :volume_type
  
  def attach_to_instance(instance_id, device_name)
    # Create block device attachment
    attachment = {
      volume_id: @volume_id,
      instance_id: instance_id,
      device: device_name,
      state: 'attaching'
    }
    # Storage system makes volume available at device path
    attachment
  end
  
  def create_snapshot(description)
    # Capture current state of all blocks
    snapshot = {
      snapshot_id: generate_id,
      volume_id: @volume_id,
      start_time: Time.now,
      state: 'pending'
    }
    # Copy-on-write mechanism preserves block state
    snapshot
  end
end

Performance Metrics

Block storage performance measures through several key metrics. IOPS (Input/Output Operations Per Second) quantifies how many block operations the system completes in one second. Throughput measures data transfer rate in MB/s or GB/s. Latency represents the time between initiating an operation and its completion.

Different workloads require different performance characteristics. Databases benefit from high IOPS for random access patterns, while data analytics applications require high throughput for sequential scans. Storage systems provision performance based on volume type and size.

Implementation Approaches

Block storage implementation varies between cloud environments, on-premises infrastructure, and local development. Each approach offers different trade-offs in performance, durability, cost, and operational complexity.

Cloud Provider Block Storage

Cloud platforms provide managed block storage services that abstract physical hardware. AWS Elastic Block Store (EBS), Azure Managed Disks, and Google Persistent Disks offer volumes with configurable performance tiers. These services handle replication, snapshots, encryption, and failure recovery automatically.

Cloud block storage attaches to virtual machine instances over network connections rather than physical cables. The network introduces some latency compared to local SSDs, but cloud providers optimize their storage networks for low latency and high throughput. Different volume types offer varying performance profiles optimized for specific workloads.

# AWS EBS volume creation and attachment
require 'aws-sdk-ec2'

ec2 = Aws::EC2::Client.new(region: 'us-east-1')

# Create a general purpose SSD volume
volume = ec2.create_volume({
  availability_zone: 'us-east-1a',
  size: 100,  # GB
  volume_type: 'gp3',
  iops: 3000,
  throughput: 125,  # MB/s
  encrypted: true
})

volume_id = volume.volume_id

# Wait for volume to become available
ec2.wait_until(:volume_available, volume_ids: [volume_id])

# Attach to EC2 instance
ec2.attach_volume({
  device: '/dev/sdf',
  instance_id: 'i-1234567890abcdef0',
  volume_id: volume_id
})

Cloud storage services offer multiple performance tiers. General purpose SSD volumes balance cost and performance for most workloads. Provisioned IOPS SSD volumes deliver consistent high performance for databases and latency-sensitive applications. Throughput-optimized HDD volumes suit big data and log processing. Cold HDD volumes provide lowest cost for infrequently accessed data.

Storage Area Networks (SAN)

Enterprise environments often deploy SAN infrastructure for centralized block storage. SAN separates storage networks from application networks, using protocols like Fibre Channel, iSCSI, or Fibre Channel over Ethernet (FCoE). Storage arrays present logical units (LUNs) that hosts mount as block devices.

SAN implementations provide advanced features including tiered storage, deduplication, compression, and synchronous replication. The storage array controller manages these features transparently to connected hosts. Administrators provision and manage storage centrally through array management interfaces.

# iSCSI initiator configuration for SAN access
class IscsiInitiator
  def discover_targets(portal_address)
    # Send discovery request to iSCSI portal
    targets = `iscsiadm -m discovery -t sendtargets -p #{portal_address}`
    parse_targets(targets)
  end
  
  def login_target(target_iqn, portal)
    # Establish connection to iSCSI target
    result = system("iscsiadm -m node -T #{target_iqn} -p #{portal} --login")
    # Target appears as /dev/sdX after successful login
    result
  end
  
  def list_sessions
    # Show active iSCSI connections
    sessions = `iscsiadm -m session`
    parse_sessions(sessions)
  end
end

Local Block Devices

Local SSDs and NVMe drives attached directly to compute instances provide lowest latency block storage. These devices eliminate network overhead but lack the durability and snapshot capabilities of network-attached storage. Applications must implement their own replication for high availability.

Local block storage suits temporary data, caching layers, and applications that replicate data at the application level. Databases like Cassandra and MongoDB distribute data across multiple nodes with local storage, achieving durability through replication rather than storage-level redundancy.

Software-Defined Storage

Software-defined storage systems like Ceph and GlusterFS create distributed block storage from commodity hardware. These systems replicate data across multiple nodes for redundancy and distribute I/O load across the cluster. The software layer presents unified block storage abstractions while managing data placement and failure recovery.

# Ceph RBD (RADOS Block Device) interaction
class CephBlockDevice
  def create_image(pool_name, image_name, size_mb)
    # Create a new block device image in Ceph
    command = "rbd create #{pool_name}/#{image_name} --size #{size_mb}"
    system(command)
  end
  
  def map_device(pool_name, image_name)
    # Map Ceph image as local block device
    command = "rbd map #{pool_name}/#{image_name}"
    device = `#{command}`.strip
    # Returns device path like /dev/rbd0
    device
  end
  
  def create_snapshot(pool_name, image_name, snap_name)
    command = "rbd snap create #{pool_name}/#{image_name}@#{snap_name}"
    system(command)
  end
end

Software-defined storage scales horizontally by adding nodes to the cluster. The system automatically rebalances data as capacity increases. This approach provides cost-effective storage at scale but requires operational expertise to maintain performance and reliability.

Ruby Implementation

Ruby applications interact with block storage through various SDKs and system interfaces. Cloud provider SDKs offer high-level APIs for managing volumes, while lower-level system calls enable direct block device access.

Cloud Provider SDKs

AWS SDK for Ruby provides comprehensive EBS volume management capabilities. Applications create, attach, detach, snapshot, and delete volumes programmatically. The SDK handles authentication, request signing, and retry logic.

require 'aws-sdk-ec2'

class VolumeManager
  def initialize(region)
    @ec2 = Aws::EC2::Client.new(region: region)
    @resource = Aws::EC2::Resource.new(region: region)
  end
  
  def create_volume(az, size_gb, volume_type: 'gp3', iops: nil, throughput: nil)
    params = {
      availability_zone: az,
      size: size_gb,
      volume_type: volume_type,
      encrypted: true,
      tag_specifications: [{
        resource_type: 'volume',
        tags: [
          { key: 'Name', value: 'AppData' },
          { key: 'Environment', value: 'Production' }
        ]
      }]
    }
    
    # Add performance parameters for gp3 volumes
    if volume_type == 'gp3'
      params[:iops] = iops || 3000
      params[:throughput] = throughput || 125
    elsif volume_type == 'io2'
      params[:iops] = iops || 1000
    end
    
    volume = @ec2.create_volume(params)
    
    # Wait for volume availability
    @ec2.wait_until(:volume_available, volume_ids: [volume.volume_id]) do |w|
      w.max_attempts = 40
      w.delay = 15
    end
    
    volume.volume_id
  end
  
  def attach_volume(volume_id, instance_id, device_name)
    @ec2.attach_volume({
      device: device_name,
      instance_id: instance_id,
      volume_id: volume_id
    })
    
    # Wait for attachment to complete
    @ec2.wait_until(:volume_in_use, volume_ids: [volume_id])
    
    # Volume now available at device_name path
    device_name
  end
  
  def create_snapshot(volume_id, description)
    snapshot = @ec2.create_snapshot({
      volume_id: volume_id,
      description: description,
      tag_specifications: [{
        resource_type: 'snapshot',
        tags: [
          { key: 'Name', value: "#{description}-#{Time.now.strftime('%Y%m%d')}" },
          { key: 'BackupType', value: 'Automated' }
        ]
      }]
    })
    
    snapshot.snapshot_id
  end
  
  def resize_volume(volume_id, new_size_gb)
    @ec2.modify_volume({
      volume_id: volume_id,
      size: new_size_gb
    })
    
    # Modification happens online without detachment
    # File system still needs to be extended after modification
  end
  
  def list_volumes(filters = {})
    volumes = @resource.volumes(filters: filters.map { |k, v| 
      { name: k.to_s, values: Array(v) }
    })
    
    volumes.map do |vol|
      {
        volume_id: vol.id,
        size: vol.size,
        volume_type: vol.volume_type,
        state: vol.state,
        iops: vol.iops,
        throughput: vol.throughput,
        attachments: vol.attachments.map { |a|
          {
            instance_id: a.instance_id,
            device: a.device,
            state: a.state
          }
        }
      }
    end
  end
end

# Usage
manager = VolumeManager.new('us-east-1')

# Create high-performance database volume
volume_id = manager.create_volume(
  'us-east-1a',
  500,
  volume_type: 'io2',
  iops: 10000
)

# Attach to database instance
manager.attach_volume(volume_id, 'i-abc123', '/dev/sdf')

# Create nightly snapshot
manager.create_snapshot(volume_id, 'Nightly database backup')

Azure SDK for Ruby provides similar functionality for managing Azure Managed Disks. Applications specify disk tier (Premium SSD, Standard SSD, Standard HDD), size, and availability zone during creation.

require 'azure_mgmt_compute'

class AzureDiskManager
  def initialize(subscription_id, tenant_id, client_id, client_secret)
    token_provider = MsRestAzure::ApplicationTokenProvider.new(
      tenant_id, client_id, client_secret
    )
    credentials = MsRest::TokenCredentials.new(token_provider)
    
    @compute_client = Azure::Compute::Profiles::Latest::Mgmt::Client.new(
      credentials: credentials,
      subscription_id: subscription_id
    )
  end
  
  def create_disk(resource_group, location, disk_name, size_gb, sku: 'Premium_LRS')
    disk_params = @compute_client.model_classes.disk.new.tap do |disk|
      disk.location = location
      disk.sku = @compute_client.model_classes.disk_sku.new.tap do |s|
        s.name = sku  # Premium_LRS, StandardSSD_LRS, Standard_LRS
      end
      disk.disk_size_gb = size_gb
      disk.creation_data = @compute_client.model_classes.creation_data.new.tap do |cd|
        cd.create_option = 'Empty'
      end
    end
    
    @compute_client.disks.create_or_update(
      resource_group,
      disk_name,
      disk_params
    ).value!
  end
  
  def attach_disk_to_vm(resource_group, vm_name, disk_id, lun)
    vm = @compute_client.virtual_machines.get(resource_group, vm_name)
    
    data_disk = @compute_client.model_classes.data_disk.new.tap do |dd|
      dd.lun = lun
      dd.create_option = 'Attach'
      dd.managed_disk = @compute_client.model_classes.managed_disk_parameters.new.tap do |md|
        md.id = disk_id
      end
    end
    
    vm.storage_profile.data_disks << data_disk
    
    @compute_client.virtual_machines.create_or_update(
      resource_group,
      vm_name,
      vm
    ).value!
  end
end

Direct Block Device Access

Ruby applications can read and write block devices directly using standard File I/O operations. This approach requires elevated permissions and careful handling of raw device data.

class BlockDeviceIO
  BLOCK_SIZE = 4096
  
  def initialize(device_path)
    @device_path = device_path
    @device = File.open(device_path, 'rb+')
  end
  
  def read_block(lba)
    @device.seek(lba * BLOCK_SIZE)
    @device.read(BLOCK_SIZE)
  end
  
  def write_block(lba, data)
    raise ArgumentError, "Data must be #{BLOCK_SIZE} bytes" unless data.bytesize == BLOCK_SIZE
    
    @device.seek(lba * BLOCK_SIZE)
    @device.write(data)
    @device.flush
  end
  
  def read_blocks(start_lba, count)
    @device.seek(start_lba * BLOCK_SIZE)
    @device.read(count * BLOCK_SIZE)
  end
  
  def size_blocks
    File.size(@device_path) / BLOCK_SIZE
  end
  
  def close
    @device.close
  end
end

# Direct database-style I/O
device = BlockDeviceIO.new('/dev/xvdf')

# Write data at block 1000
data = "A" * 4096
device.write_block(1000, data)

# Read it back
retrieved = device.read_block(1000)
puts retrieved == data  # => true

device.close

File System Management

Ruby scripts often automate file system creation and mounting on attached block volumes. The sys-filesystem gem provides cross-platform file system information.

require 'sys/filesystem'

class FileSystemManager
  def format_device(device_path, fs_type: 'ext4', label: nil)
    command = ['mkfs', "-t #{fs_type}"]
    command << "-L #{label}" if label
    command << device_path
    
    result = system(command.join(' '))
    raise "Format failed for #{device_path}" unless result
  end
  
  def mount_device(device_path, mount_point, fs_type: 'ext4', options: 'defaults')
    Dir.mkdir(mount_point) unless Dir.exist?(mount_point)
    
    command = "mount -t #{fs_type} -o #{options} #{device_path} #{mount_point}"
    result = system(command)
    raise "Mount failed for #{device_path}" unless result
    
    mount_point
  end
  
  def get_filesystem_stats(mount_point)
    stat = Sys::Filesystem.stat(mount_point)
    
    {
      total_bytes: stat.blocks * stat.block_size,
      available_bytes: stat.blocks_available * stat.block_size,
      free_bytes: stat.blocks_free * stat.block_size,
      used_bytes: (stat.blocks - stat.blocks_free) * stat.block_size,
      percent_used: ((stat.blocks - stat.blocks_free).to_f / stat.blocks * 100).round(2)
    }
  end
  
  def add_to_fstab(device_path, mount_point, fs_type, options = 'defaults', dump = 0, pass = 2)
    fstab_entry = "#{device_path} #{mount_point} #{fs_type} #{options} #{dump} #{pass}\n"
    
    File.open('/etc/fstab', 'a') do |f|
      f.write(fstab_entry)
    end
  end
end

# Automate volume setup
fs_manager = FileSystemManager.new

# Format newly attached volume
fs_manager.format_device('/dev/xvdf', fs_type: 'xfs', label: 'AppData')

# Mount it
fs_manager.mount_device('/dev/xvdf', '/mnt/data', fs_type: 'xfs')

# Make mount persistent across reboots
fs_manager.add_to_fstab('/dev/xvdf', '/mnt/data', 'xfs')

# Check usage
stats = fs_manager.get_filesystem_stats('/mnt/data')
puts "Used: #{stats[:percent_used]}%"

Snapshot Management Automation

Ruby scripts automate snapshot lifecycle management, including creation, retention policies, and cross-region copying.

class SnapshotManager
  def initialize(ec2_client)
    @ec2 = ec2_client
  end
  
  def create_tagged_snapshot(volume_id, retention_days)
    snapshot = @ec2.create_snapshot({
      volume_id: volume_id,
      description: "Automated backup #{Time.now.iso8601}",
      tag_specifications: [{
        resource_type: 'snapshot',
        tags: [
          { key: 'AutomatedBackup', value: 'true' },
          { key: 'RetentionDays', value: retention_days.to_s },
          { key: 'DeleteAfter', value: (Time.now + retention_days * 86400).iso8601 }
        ]
      }]
    })
    
    snapshot.snapshot_id
  end
  
  def cleanup_expired_snapshots
    snapshots = @ec2.describe_snapshots({
      owner_ids: ['self'],
      filters: [{ name: 'tag:AutomatedBackup', values: ['true'] }]
    }).snapshots
    
    deleted_count = 0
    
    snapshots.each do |snapshot|
      delete_after_tag = snapshot.tags.find { |t| t.key == 'DeleteAfter' }
      next unless delete_after_tag
      
      delete_after = Time.parse(delete_after_tag.value)
      
      if Time.now > delete_after
        @ec2.delete_snapshot(snapshot_id: snapshot.snapshot_id)
        deleted_count += 1
      end
    end
    
    deleted_count
  end
  
  def copy_snapshot_to_region(snapshot_id, destination_region, description)
    destination_ec2 = Aws::EC2::Client.new(region: destination_region)
    
    destination_ec2.copy_snapshot({
      source_region: @ec2.config.region,
      source_snapshot_id: snapshot_id,
      description: description,
      encrypted: true
    })
  end
end

Tools & Ecosystem

Block storage management spans cloud provider tools, open source systems, and monitoring utilities. Each category addresses different aspects of storage operations.

Cloud Provider Management Tools

AWS provides the AWS CLI for command-line volume operations and the AWS Management Console for graphical management. The AWS CLI enables scriptable workflows for volume lifecycle management.

# Ruby wrapper for AWS CLI operations
class AwsCliWrapper
  def describe_volumes(region, filters = {})
    filter_args = filters.map { |k, v| "--filters Name=#{k},Values=#{v}" }.join(' ')
    command = "aws ec2 describe-volumes --region #{region} #{filter_args} --output json"
    
    output = `#{command}`
    JSON.parse(output)
  end
  
  def wait_for_snapshot(region, snapshot_id)
    command = "aws ec2 wait snapshot-completed --region #{region} --snapshot-ids #{snapshot_id}"
    system(command)
  end
end

Azure offers Azure CLI and Azure Portal for disk management. Google Cloud provides gcloud CLI and Cloud Console. These tools support infrastructure-as-code workflows when combined with automation scripts.

Infrastructure as Code Tools

Terraform manages block storage resources declaratively across multiple cloud providers. Ruby applications generate Terraform configurations programmatically for dynamic infrastructure provisioning.

class TerraformVolumeGenerator
  def generate_ebs_volume_config(name, size_gb, volume_type, iops = nil)
    config = <<~HCL
      resource "aws_ebs_volume" "#{name}" {
        availability_zone = var.availability_zone
        size              = #{size_gb}
        type              = "#{volume_type}"
    HCL
    
    config += "  iops              = #{iops}\n" if iops
    config += "  encrypted         = true\n"
    config += "  tags = {\n"
    config += "    Name = \"#{name}\"\n"
    config += "  }\n"
    config += "}\n"
    
    config
  end
  
  def generate_volume_attachment(volume_resource, instance_id, device_name)
    <<~HCL
      resource "aws_volume_attachment" "#{volume_resource}_attachment" {
        device_name = "#{device_name}"
        volume_id   = aws_ebs_volume.#{volume_resource}.id
        instance_id = "#{instance_id}"
      }
    HCL
  end
end

generator = TerraformVolumeGenerator.new
puts generator.generate_ebs_volume_config('database_volume', 500, 'io2', 5000)
puts generator.generate_volume_attachment('database_volume', 'aws_instance.db.id', '/dev/sdf')

Open Source Storage Systems

Ceph provides distributed block storage through its RBD (RADOS Block Device) component. Ruby applications interact with Ceph through command-line tools or API bindings.

class CephRbdManager
  def create_rbd_image(pool, image_name, size_mb, features: ['layering'])
    feature_args = features.map { |f| "--image-feature #{f}" }.join(' ')
    command = "rbd create #{pool}/#{image_name} --size #{size_mb} #{feature_args}"
    system(command)
  end
  
  def map_image(pool, image_name)
    command = "rbd map #{pool}/#{image_name}"
    device = `#{command}`.strip
    device
  end
  
  def resize_image(pool, image_name, new_size_mb)
    command = "rbd resize #{pool}/#{image_name} --size #{new_size_mb}"
    system(command)
  end
  
  def create_snapshot(pool, image_name, snap_name)
    command = "rbd snap create #{pool}/#{image_name}@#{snap_name}"
    system(command)
  end
  
  def protect_snapshot(pool, image_name, snap_name)
    command = "rbd snap protect #{pool}/#{image_name}@#{snap_name}"
    system(command)
  end
  
  def clone_from_snapshot(pool, image_name, snap_name, new_pool, new_image_name)
    command = "rbd clone #{pool}/#{image_name}@#{snap_name} #{new_pool}/#{new_image_name}"
    system(command)
  end
end

GlusterFS provides distributed file storage built on block devices. OpenStack Cinder offers block storage services for private clouds with multiple storage backend support.

Monitoring and Performance Tools

iostat monitors block device I/O statistics, displaying IOPS, throughput, and latency metrics. Ruby applications parse iostat output for performance analysis.

class IoStatMonitor
  def parse_iostat_output(device)
    output = `iostat -x #{device} 1 2`
    lines = output.split("\n")
    
    # Find device statistics line (second occurrence after header)
    stats_line = lines.reverse.find { |l| l.start_with?(device) }
    return nil unless stats_line
    
    parts = stats_line.split
    
    {
      device: parts[0],
      rrqm_per_sec: parts[1].to_f,
      wrqm_per_sec: parts[2].to_f,
      r_per_sec: parts[3].to_f,
      w_per_sec: parts[4].to_f,
      rkb_per_sec: parts[5].to_f,
      wkb_per_sec: parts[6].to_f,
      avg_request_size_kb: parts[7].to_f,
      avg_queue_length: parts[8].to_f,
      await_ms: parts[9].to_f,
      r_await_ms: parts[10].to_f,
      w_await_ms: parts[11].to_f,
      percent_util: parts[13].to_f
    }
  end
  
  def monitor_device(device, duration_seconds)
    stats = []
    (duration_seconds / 5).times do
      stat = parse_iostat_output(device)
      stats << stat if stat
      sleep 5
    end
    stats
  end
  
  def calculate_averages(stats)
    return {} if stats.empty?
    
    avg = {}
    stats.first.keys.each do |key|
      next if key == :device
      values = stats.map { |s| s[key] }
      avg[key] = values.sum / values.size.to_f
    end
    avg
  end
end

monitor = IoStatMonitor.new
stats = monitor.monitor_device('xvdf', 60)
averages = monitor.calculate_averages(stats)
puts "Average IOPS: #{averages[:r_per_sec] + averages[:w_per_sec]}"
puts "Average latency: #{averages[:await_ms]}ms"

CloudWatch metrics track EBS volume performance in AWS environments. Applications query CloudWatch for volume IOPS, throughput, and burst balance metrics.

Ruby Gems for Storage Management

The aws-sdk-ec2 gem provides the primary interface for AWS block storage. The fog gem offers multi-cloud storage abstraction, supporting AWS, Azure, Google Cloud, and OpenStack.

require 'fog/aws'

compute = Fog::Compute.new({
  provider: 'AWS',
  region: 'us-east-1',
  aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
  aws_secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
})

# Create volume using Fog
volume = compute.volumes.create({
  availability_zone: 'us-east-1a',
  size: 100,
  volume_type: 'gp3'
})

volume.wait_for { ready? }

# Attach volume
server = compute.servers.get('i-1234567890abcdef0')
volume.server = server
volume.device = '/dev/sdf'
volume.save

Performance Considerations

Block storage performance depends on hardware characteristics, workload patterns, and configuration choices. Applications must understand these factors to achieve optimal performance.

IOPS and Throughput Characteristics

Different block storage types deliver different performance profiles. Provisioned IOPS SSD volumes guarantee specific IOPS levels, while general purpose SSD volumes use credit-based systems that allow bursting above baseline performance. HDD volumes optimize for throughput rather than IOPS.

class VolumePerformanceCalculator
  # AWS gp3 baseline performance
  GP3_BASE_IOPS = 3000
  GP3_BASE_THROUGHPUT = 125  # MB/s
  GP3_MAX_IOPS = 16000
  GP3_MAX_THROUGHPUT = 1000  # MB/s
  
  # Performance per provisioned GB
  IO2_IOPS_PER_GB = 500
  IO2_MAX_IOPS = 64000
  
  def calculate_gp3_performance(size_gb, provisioned_iops = nil, provisioned_throughput = nil)
    iops = provisioned_iops || GP3_BASE_IOPS
    throughput = provisioned_throughput || GP3_BASE_THROUGHPUT
    
    {
      volume_type: 'gp3',
      size_gb: size_gb,
      iops: [iops, GP3_MAX_IOPS].min,
      throughput_mb_s: [throughput, GP3_MAX_THROUGHPUT].min,
      max_io_size_kb: (throughput * 1024.0 / iops).round(2)
    }
  end
  
  def calculate_io2_performance(size_gb, provisioned_iops)
    max_allowed_iops = size_gb * IO2_IOPS_PER_GB
    actual_iops = [provisioned_iops, max_allowed_iops, IO2_MAX_IOPS].min
    
    {
      volume_type: 'io2',
      size_gb: size_gb,
      iops: actual_iops,
      throughput_mb_s: (actual_iops * 256.0 / 1024).round(2),  # Assumes 256KB I/O
      iops_per_gb_ratio: (actual_iops.to_f / size_gb).round(2)
    }
  end
  
  def recommend_volume_type(workload_profile)
    case workload_profile[:access_pattern]
    when :random_read_heavy
      if workload_profile[:required_iops] > 16000
        { type: 'io2', reason: 'High IOPS requirement exceeds gp3 limits' }
      else
        { type: 'gp3', reason: 'Cost-effective for moderate IOPS workloads' }
      end
    when :sequential_scan
      if workload_profile[:required_throughput_mb_s] > 1000
        { type: 'io2', reason: 'Very high throughput requirement' }
      else
        { type: 'gp3', reason: 'Adequate throughput for sequential workloads' }
      end
    when :mixed
      { type: 'gp3', reason: 'Balanced performance for mixed workloads' }
    else
      { type: 'gp3', reason: 'Default choice for unknown patterns' }
    end
  end
end

calculator = VolumePerformanceCalculator.new

# Calculate performance for database volume
perf = calculator.calculate_io2_performance(500, 10000)
puts "Database volume: #{perf[:iops]} IOPS, #{perf[:throughput_mb_s]} MB/s"

# Get recommendation
recommendation = calculator.recommend_volume_type({
  access_pattern: :random_read_heavy,
  required_iops: 5000,
  required_throughput_mb_s: 200
})
puts "Recommended: #{recommendation[:type]} - #{recommendation[:reason]}"

Access Pattern Optimization

Sequential access patterns achieve higher throughput than random access. Applications that read or write data in contiguous blocks benefit from read-ahead caching and write buffering. Random access patterns require higher IOPS capacity to maintain throughput.

Database query performance depends heavily on index organization and query patterns. Queries that scan large ranges perform better with sequential I/O optimization, while point lookups benefit from high IOPS configurations.

class IoPatternAnalyzer
  def analyze_access_pattern(operations)
    sorted_ops = operations.sort_by { |op| op[:lba] }
    
    sequential_count = 0
    random_count = 0
    
    sorted_ops.each_cons(2) do |op1, op2|
      if (op2[:lba] - op1[:lba]).abs <= 8  # Within 8 blocks
        sequential_count += 1
      else
        random_count += 1
      end
    end
    
    total = sequential_count + random_count
    sequential_ratio = total > 0 ? sequential_count.to_f / total : 0
    
    {
      total_operations: operations.size,
      sequential_operations: sequential_count,
      random_operations: random_count,
      sequential_ratio: sequential_ratio,
      pattern_type: classify_pattern(sequential_ratio)
    }
  end
  
  def classify_pattern(sequential_ratio)
    case sequential_ratio
    when 0.8..1.0
      :highly_sequential
    when 0.5..0.8
      :mostly_sequential
    when 0.2..0.5
      :mixed
    else
      :highly_random
    end
  end
  
  def suggest_optimizations(pattern_type)
    case pattern_type
    when :highly_sequential
      [
        'Increase read-ahead cache',
        'Use larger I/O sizes',
        'Consider throughput-optimized volumes'
      ]
    when :mostly_sequential
      [
        'Enable moderate read-ahead',
        'Balance between IOPS and throughput provisioning'
      ]
    when :mixed
      [
        'Profile both IOPS and throughput requirements',
        'Consider gp3 with custom provisioning'
      ]
    when :highly_random
      [
        'Provision high IOPS capacity',
        'Disable excessive read-ahead',
        'Consider io2 volumes for consistent performance'
      ]
    end
  end
end

Latency Factors

Network latency affects cloud block storage performance. Typical network-attached storage introduces 1-5ms latency per operation, while local NVMe SSDs achieve sub-millisecond latency. Applications sensitive to latency benefit from local instance storage or high-performance provisioned IOPS volumes.

Queue depth affects achievable IOPS. Higher queue depths enable parallel I/O operations, increasing total IOPS but potentially increasing individual operation latency. Applications must balance queue depth to optimize for their latency requirements.

File System Impact

File system choice affects block storage performance. XFS handles large files and parallel I/O well. Ext4 provides good general-purpose performance with extensive tooling support. Both file systems add overhead above raw block device performance.

class FileSystemBenchmark
  def benchmark_write_performance(mount_point, file_size_mb, block_size_kb)
    test_file = File.join(mount_point, 'benchmark_test')
    block_size_bytes = block_size_kb * 1024
    blocks = (file_size_mb * 1024 * 1024) / block_size_bytes
    data = 'X' * block_size_bytes
    
    start_time = Time.now
    
    File.open(test_file, 'wb') do |f|
      blocks.times do
        f.write(data)
      end
      f.flush
      f.fsync  # Force sync to disk
    end
    
    duration = Time.now - start_time
    throughput_mb_s = file_size_mb / duration
    
    File.delete(test_file)
    
    {
      file_size_mb: file_size_mb,
      block_size_kb: block_size_kb,
      duration_seconds: duration.round(2),
      throughput_mb_s: throughput_mb_s.round(2)
    }
  end
  
  def benchmark_read_performance(mount_point, file_size_mb, block_size_kb)
    test_file = File.join(mount_point, 'benchmark_test')
    block_size_bytes = block_size_kb * 1024
    
    # Create test file
    File.open(test_file, 'wb') do |f|
      f.write('X' * (file_size_mb * 1024 * 1024))
    end
    
    # Clear page cache (requires root)
    system('sync; echo 3 > /proc/sys/vm/drop_caches') rescue nil
    
    start_time = Time.now
    
    File.open(test_file, 'rb') do |f|
      while chunk = f.read(block_size_bytes)
        # Read entire file
      end
    end
    
    duration = Time.now - start_time
    throughput_mb_s = file_size_mb / duration
    
    File.delete(test_file)
    
    {
      file_size_mb: file_size_mb,
      block_size_kb: block_size_kb,
      duration_seconds: duration.round(2),
      throughput_mb_s: throughput_mb_s.round(2)
    }
  end
end

Caching Strategies

Operating system page cache significantly improves read performance for frequently accessed data. Applications can use direct I/O to bypass page cache when testing raw storage performance or when managing caching at the application level.

Database systems implement their own buffer pools to cache frequently accessed blocks. These application-level caches operate above the file system, providing application-specific optimization opportunities.

Integration & Interoperability

Block storage integrates with applications, backup systems, and disaster recovery infrastructure. Understanding integration patterns enables robust storage architectures.

Application Integration Patterns

Databases mount block volumes as data directories, storing table files and transaction logs on persistent storage. The database manages its own I/O patterns and caching, interacting directly with the file system layer.

class DatabaseVolumeSetup
  def initialize(volume_manager, fs_manager)
    @volume_manager = volume_manager
    @fs_manager = fs_manager
  end
  
  def setup_postgresql_storage(instance_id, data_size_gb, wal_size_gb)
    # Create separate volumes for data and WAL
    data_volume_id = @volume_manager.create_volume(
      'us-east-1a',
      data_size_gb,
      volume_type: 'gp3',
      iops: 5000,
      throughput: 250
    )
    
    wal_volume_id = @volume_manager.create_volume(
      'us-east-1a',
      wal_size_gb,
      volume_type: 'io2',
      iops: 3000  # WAL benefits from low latency
    )
    
    # Attach volumes
    @volume_manager.attach_volume(data_volume_id, instance_id, '/dev/sdf')
    @volume_manager.attach_volume(wal_volume_id, instance_id, '/dev/sdg')
    
    # Format and mount
    @fs_manager.format_device('/dev/sdf', fs_type: 'xfs', label: 'pgdata')
    @fs_manager.format_device('/dev/sdg', fs_type: 'xfs', label: 'pgwal')
    
    @fs_manager.mount_device('/dev/sdf', '/var/lib/postgresql/data', fs_type: 'xfs')
    @fs_manager.mount_device('/dev/sdg', '/var/lib/postgresql/wal', fs_type: 'xfs')
    
    # Make persistent
    @fs_manager.add_to_fstab('/dev/sdf', '/var/lib/postgresql/data', 'xfs')
    @fs_manager.add_to_fstab('/dev/sdg', '/var/lib/postgresql/wal', 'xfs')
    
    {
      data_volume: data_volume_id,
      wal_volume: wal_volume_id,
      data_mount: '/var/lib/postgresql/data',
      wal_mount: '/var/lib/postgresql/wal'
    }
  end
end

Container orchestration platforms like Kubernetes use persistent volumes backed by block storage. The Container Storage Interface (CSI) provides a standard API for container runtimes to provision and attach block storage to containers.

Backup and Recovery Integration

Snapshot-based backups capture point-in-time copies of block volumes. Applications coordinate snapshots with database checkpoints or application quiescing to ensure consistency.

class BackupCoordinator
  def initialize(snapshot_manager, notification_service)
    @snapshot_manager = snapshot_manager
    @notification_service = notification_service
  end
  
  def create_application_consistent_backup(volume_id, application_type)
    begin
      # Flush application buffers
      flush_application(application_type)
      
      # Create snapshot
      snapshot_id = @snapshot_manager.create_tagged_snapshot(volume_id, 30)
      
      # Resume normal operations
      resume_application(application_type)
      
      @notification_service.send_success(
        "Backup created: #{snapshot_id}",
        volume_id: volume_id
      )
      
      snapshot_id
    rescue StandardError => e
      resume_application(application_type) rescue nil
      @notification_service.send_failure(
        "Backup failed: #{e.message}",
        volume_id: volume_id
      )
      raise
    end
  end
  
  def flush_application(app_type)
    case app_type
    when :postgresql
      system('sudo -u postgres psql -c "CHECKPOINT;"')
    when :mysql
      system('mysql -e "FLUSH TABLES WITH READ LOCK;"')
    when :mongodb
      system('mongo admin --eval "db.fsyncLock()"')
    end
  end
  
  def resume_application(app_type)
    case app_type
    when :mysql
      system('mysql -e "UNLOCK TABLES;"')
    when :mongodb
      system('mongo admin --eval "db.fsyncUnlock()"')
    end
  end
  
  def restore_from_snapshot(snapshot_id, target_instance_id, device_name)
    # Create volume from snapshot
    ec2 = Aws::EC2::Client.new
    
    volume = ec2.create_volume({
      snapshot_id: snapshot_id,
      availability_zone: get_instance_az(target_instance_id),
      volume_type: 'gp3'
    })
    
    # Wait for volume
    ec2.wait_until(:volume_available, volume_ids: [volume.volume_id])
    
    # Attach to instance
    @volume_manager.attach_volume(volume.volume_id, target_instance_id, device_name)
    
    volume.volume_id
  end
  
  def get_instance_az(instance_id)
    ec2 = Aws::EC2::Client.new
    instance = ec2.describe_instances(instance_ids: [instance_id]).reservations.first.instances.first
    instance.placement.availability_zone
  end
end

Replication and Disaster Recovery

Cross-region replication protects against regional failures. Applications copy snapshots to secondary regions for disaster recovery.

class DisasterRecoveryManager
  def initialize(primary_region, dr_region)
    @primary_ec2 = Aws::EC2::Client.new(region: primary_region)
    @dr_ec2 = Aws::EC2::Client.new(region: dr_region)
    @primary_region = primary_region
    @dr_region = dr_region
  end
  
  def replicate_snapshot_to_dr(snapshot_id, description)
    # Copy snapshot to DR region
    copy_result = @dr_ec2.copy_snapshot({
      source_region: @primary_region,
      source_snapshot_id: snapshot_id,
      description: "DR copy: #{description}",
      encrypted: true
    })
    
    # Tag DR snapshot
    @dr_ec2.create_tags({
      resources: [copy_result.snapshot_id],
      tags: [
        { key: 'DisasterRecovery', value: 'true' },
        { key: 'SourceRegion', value: @primary_region },
        { key: 'SourceSnapshot', value: snapshot_id }
      ]
    })
    
    copy_result.snapshot_id
  end
  
  def setup_continuous_replication(volume_id, schedule_expression)
    # Create EventBridge rule for automated replication
    rule_name = "dr-replication-#{volume_id}"
    
    {
      rule_name: rule_name,
      schedule: schedule_expression,
      target: {
        action: 'replicate_snapshot',
        volume_id: volume_id
      }
    }
  end
  
  def failover_to_dr(primary_volume_id)
    # Find latest DR snapshot
    snapshots = @dr_ec2.describe_snapshots({
      owner_ids: ['self'],
      filters: [
        { name: 'tag:SourceVolume', values: [primary_volume_id] }
      ]
    }).snapshots.sort_by(&:start_time).reverse
    
    latest_snapshot = snapshots.first
    raise "No DR snapshots found for #{primary_volume_id}" unless latest_snapshot
    
    # Create volume in DR region
    volume = @dr_ec2.create_volume({
      snapshot_id: latest_snapshot.snapshot_id,
      availability_zone: "#{@dr_region}a",
      volume_type: 'gp3'
    })
    
    {
      dr_volume_id: volume.volume_id,
      snapshot_id: latest_snapshot.snapshot_id,
      snapshot_age: Time.now - latest_snapshot.start_time
    }
  end
end

Monitoring Integration

CloudWatch provides detailed metrics for EBS volumes including VolumeReadOps, VolumeWriteOps, VolumeReadBytes, VolumeWriteBytes, and VolumeThroughputPercentage. Applications query these metrics to monitor storage health.

require 'aws-sdk-cloudwatch'

class VolumeMonitoring
  def initialize(region)
    @cloudwatch = Aws::CloudWatch::Client.new(region: region)
  end
  
  def get_volume_metrics(volume_id, start_time, end_time)
    metrics = {}
    
    metric_names = [
      'VolumeReadOps',
      'VolumeWriteOps',
      'VolumeReadBytes',
      'VolumeWriteBytes',
      'VolumeThroughputPercentage',
      'VolumeConsumedReadWriteOps'
    ]
    
    metric_names.each do |metric_name|
      stats = @cloudwatch.get_metric_statistics({
        namespace: 'AWS/EBS',
        metric_name: metric_name,
        dimensions: [{ name: 'VolumeId', value: volume_id }],
        start_time: start_time,
        end_time: end_time,
        period: 300,  # 5 minutes
        statistics: ['Average', 'Maximum', 'Sum']
      })
      
      metrics[metric_name] = stats.datapoints.map { |dp|
        {
          timestamp: dp.timestamp,
          average: dp.average,
          maximum: dp.maximum,
          sum: dp.sum
        }
      }
    end
    
    metrics
  end
  
  def calculate_iops(volume_id, start_time, end_time)
    read_ops = get_volume_metrics(volume_id, start_time, end_time)['VolumeReadOps']
    write_ops = get_volume_metrics(volume_id, start_time, end_time)['VolumeWriteOps']
    
    combined = read_ops.zip(write_ops).map do |r, w|
      {
        timestamp: r[:timestamp],
        read_iops: r[:sum] / 300.0,  # Convert to per-second
        write_iops: w[:sum] / 300.0,
        total_iops: (r[:sum] + w[:sum]) / 300.0
      }
    end
    
    combined
  end
  
  def create_alarm(volume_id, metric_name, threshold, comparison_operator)
    alarm_name = "#{volume_id}-#{metric_name}-alarm"
    
    @cloudwatch.put_metric_alarm({
      alarm_name: alarm_name,
      alarm_description: "Alert when #{metric_name} #{comparison_operator} #{threshold}",
      actions_enabled: true,
      metric_name: metric_name,
      namespace: 'AWS/EBS',
      statistic: 'Average',
      dimensions: [{ name: 'VolumeId', value: volume_id }],
      period: 300,
      evaluation_periods: 2,
      threshold: threshold,
      comparison_operator: comparison_operator
    })
    
    alarm_name
  end
end

Reference

Volume Types Comparison

Block Storage Operations

Performance Metrics

AWS CLI Commands

File System Options

Ruby SDK Methods

Storage Architecture Patterns

Common Block Sizes

Error Codes and Handling