HTTP Headers

Overview

HTTP headers consist of key-value pairs transmitted between clients and servers during HTTP communication. Each header conveys metadata about the request or response, enabling functionality beyond the basic message body transmission. Headers control caching behavior, specify content types, manage authentication, define encoding preferences, and coordinate various aspects of client-server interaction.

The HTTP specification divides headers into four categories: general headers applicable to both requests and responses, request headers sent from client to server, response headers returned from server to client, and entity headers describing the message body. This categorization reflects the bidirectional nature of HTTP communication and the distinct roles each participant plays.

Headers operate at the application layer of the network stack, positioned between the HTTP method/status line and the message body. Servers and clients parse headers before processing body content, making header data available for routing decisions, authentication checks, content negotiation, and other preprocessing operations.

# Basic header structure in HTTP
GET /api/users HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0
Accept: application/json
Authorization: Bearer token123

# Response with headers
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: max-age=3600
Set-Cookie: session=abc123

Key Principles

HTTP headers follow a structured format where each header consists of a case-insensitive name, a colon separator, and a value. Multiple headers with the same name can appear in a single message, and header order generally does not affect semantics except for certain precedence rules in specific headers.

Request Headers originate from the client and inform the server about the client's capabilities, preferences, and context. The Host header identifies the target server, essential for virtual hosting scenarios where multiple domains share a single IP address. User-Agent reveals the client software making the request, though this data often proves unreliable due to spoofing. Accept headers negotiate content types, languages, and encodings the client can process. Authorization headers carry authentication credentials in various formats.

Response Headers flow from server to client, describing the server's capabilities, the resource's characteristics, and instructions for handling the response. Content-Type declares the media type of the response body using MIME types. Content-Length specifies body size in bytes, enabling connection reuse and progress indicators. Cache-Control directs caching behavior at intermediate proxies and client caches. Set-Cookie establishes stateful sessions over the stateless HTTP protocol.

General Headers apply to both requests and responses. Connection controls whether the network connection stays open after the current transaction completes. Date timestamps the message generation time. Via tracks intermediate proxies in the request/response chain, revealing routing paths and proxy transformations.

Entity Headers describe the message body's properties regardless of message direction. Content-Encoding indicates compression algorithms applied to the body. Content-Language specifies the natural language of the content. Content-Location provides an alternative URL for the resource when it differs from the request URL.

Header parsing requires handling multiple edge cases. Whitespace around the colon separator gets trimmed. Values can span multiple lines using continuation syntax with leading whitespace, though this practice has fallen out of favor in HTTP/2. Headers containing non-ASCII characters require encoding schemes like RFC 2047 for compatibility.

# Header continuation (deprecated in HTTP/2)
Subject: This is a long subject line
 that continues on the next line

# Properly encoded non-ASCII header
Subject: =?utf-8?B?SGVsbG8gV29ybGQ=?=

Ruby Implementation

Ruby provides multiple layers of abstraction for working with HTTP headers. The standard library includes Net::HTTP for low-level HTTP operations, while Rack standardizes the interface between web servers and Ruby applications. Higher-level frameworks like Rails build additional conveniences on these foundations.

Net::HTTP exposes headers through bracket notation on request and response objects. Setting headers modifies the outgoing request before transmission, while reading headers accesses data from received responses. Header names get normalized to standard capitalization automatically.

require 'net/http'

uri = URI('https://api.example.com/data')
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true

request = Net::HTTP::Get.new(uri)
request['Accept'] = 'application/json'
request['User-Agent'] = 'MyApp/1.0'
request['Authorization'] = 'Bearer token123'

response = http.request(request)
content_type = response['Content-Type']
cache_control = response['Cache-Control']
# => "max-age=3600, public"

Rack normalizes header handling across different Ruby web servers by converting header names to uppercase with HTTP_ prefixes and replacing hyphens with underscores. This transformation enables consistent access to headers regardless of the underlying server implementation. Special headers like Content-Type and Content-Length omit the HTTP_ prefix to distinguish them from general headers.

# Rack environment hash
env = {
  'HTTP_ACCEPT' => 'application/json',
  'HTTP_USER_AGENT' => 'Mozilla/5.0',
  'CONTENT_TYPE' => 'application/x-www-form-urlencoded',
  'CONTENT_LENGTH' => '42'
}

# Accessing headers in Rack middleware
class HeaderLogger
  def initialize(app)
    @app = app
  end
  
  def call(env)
    accept = env['HTTP_ACCEPT']
    content_type = env['CONTENT_TYPE']
    
    status, headers, body = @app.call(env)
    
    # Set response headers
    headers['X-Processing-Time'] = '0.123'
    headers['X-Server-Version'] = '1.0.0'
    
    [status, headers, body]
  end
end

Rails adds convenience methods for common header operations through the ActionDispatch framework. The request object provides typed accessors for frequently used headers, handling parsing and type conversion automatically. The response object offers methods for setting headers with validation and formatting.

# Rails controller header access
class ApiController < ApplicationController
  def index
    # Reading request headers
    accept_language = request.headers['Accept-Language']
    auth_token = request.headers['Authorization']
    
    # Using convenience methods
    content_type = request.content_type
    user_agent = request.user_agent
    
    # Setting response headers
    response.headers['X-API-Version'] = '2.0'
    response.headers['X-Rate-Limit'] = '100'
    
    # Using typed setters
    response.content_type = 'application/json'
    response.cache_control[:public] = true
    response.cache_control[:max_age] = 3600
    
    render json: data
  end
end

Faraday provides a higher-level HTTP client with middleware support for common header patterns. Middleware can inject headers automatically, log header values, or transform headers based on conditions. This architecture separates header management from business logic.

require 'faraday'

conn = Faraday.new(url: 'https://api.example.com') do |f|
  f.headers['User-Agent'] = 'MyApp/1.0'
  f.headers['Accept'] = 'application/json'
  
  f.request :authorization, 'Bearer', -> { fetch_token }
  f.request :retry, max: 3
  
  f.response :logger
  f.adapter Faraday.default_adapter
end

response = conn.get('/users') do |req|
  req.headers['X-Request-ID'] = SecureRandom.uuid
end

Security Implications

HTTP headers serve as critical security controls, though improper configuration creates vulnerabilities. Security headers instruct browsers to enforce protections that mitigate common web attacks. Missing or misconfigured security headers rank among the top web application security risks.

Content-Security-Policy (CSP) prevents cross-site scripting (XSS) attacks by specifying which sources can load JavaScript, styles, images, and other resources. A strict CSP blocks inline scripts and restricts external script sources to whitelisted domains. Implementing CSP requires auditing all script sources in an application and updating legacy inline event handlers.

# Rails security header configuration
class ApplicationController < ActionController::Base
  before_action :set_security_headers
  
  private
  
  def set_security_headers
    response.headers['Content-Security-Policy'] = [
      "default-src 'self'",
      "script-src 'self' https://cdn.example.com",
      "style-src 'self' 'unsafe-inline'",
      "img-src 'self' data: https:",
      "font-src 'self' https://fonts.gstatic.com",
      "connect-src 'self' https://api.example.com",
      "frame-ancestors 'none'"
    ].join('; ')
    
    response.headers['X-Frame-Options'] = 'DENY'
    response.headers['X-Content-Type-Options'] = 'nosniff'
    response.headers['X-XSS-Protection'] = '1; mode=block'
    response.headers['Referrer-Policy'] = 'strict-origin-when-cross-origin'
  end
end

Strict-Transport-Security (HSTS) forces HTTPS connections for a specified duration, preventing downgrade attacks and cookie hijacking. Once a browser receives an HSTS header, it refuses to connect over HTTP for the specified period, even if the user types http:// in the address bar. The includeSubDomains directive extends this protection to all subdomains.

Cross-Origin Resource Sharing (CORS) controls which origins can access resources through JavaScript. Without proper CORS headers, browsers block cross-origin requests to prevent malicious sites from stealing data. Configuring CORS requires balancing security with legitimate cross-origin needs.

# CORS configuration in Rails
class ApplicationController < ActionController::Base
  before_action :set_cors_headers
  
  private
  
  def set_cors_headers
    allowed_origins = ['https://app.example.com', 'https://admin.example.com']
    origin = request.headers['Origin']
    
    if allowed_origins.include?(origin)
      response.headers['Access-Control-Allow-Origin'] = origin
      response.headers['Access-Control-Allow-Methods'] = 'GET, POST, PUT, DELETE, OPTIONS'
      response.headers['Access-Control-Allow-Headers'] = 'Content-Type, Authorization'
      response.headers['Access-Control-Max-Age'] = '3600'
      response.headers['Access-Control-Allow-Credentials'] = 'true'
    end
  end
end

Authentication headers transmit credentials from client to server. The Authorization header carries tokens, API keys, or encoded username/password combinations. Different authentication schemes use different formats: Basic authentication encodes credentials in base64, Bearer tokens pass opaque tokens, and Digest authentication uses challenge-response mechanisms. Exposing credentials in headers requires HTTPS to prevent interception.

Cookie handling introduces multiple security concerns. The Secure flag restricts cookie transmission to HTTPS connections. HttpOnly prevents JavaScript access to cookies, mitigating XSS-based cookie theft. SameSite controls when browsers send cookies with cross-site requests, defending against CSRF attacks.

# Secure cookie configuration in Rails
Rails.application.config.session_store :cookie_store,
  key: '_app_session',
  secure: Rails.env.production?,
  httponly: true,
  same_site: :lax

# Setting cookies with security flags
cookies.signed[:user_id] = {
  value: user.id,
  expires: 1.day.from_now,
  secure: true,
  httponly: true,
  same_site: :strict
}

Header injection attacks occur when applications include unsanitized user input in headers. Attackers inject newline characters to add arbitrary headers or modify response behavior. Modern frameworks automatically sanitize header values, but manual header construction requires careful validation.

Common Patterns

Cache-Control patterns optimize performance by specifying how responses can be cached. The max-age directive sets cache lifetime in seconds. The public directive allows caching in shared caches like CDNs, while private restricts caching to client-specific caches. The no-store directive prevents all caching for sensitive data.

# Cache control patterns in Rails
class PublicController < ApplicationController
  def index
    # Cache for 1 hour in any cache
    expires_in 1.hour, public: true
    
    # Equivalent to:
    # response.cache_control[:public] = true
    # response.cache_control[:max_age] = 3600
    
    render json: public_data
  end
end

class PrivateController < ApplicationController
  def show
    # Cache in browser only, revalidate
    expires_in 5.minutes, public: false, must_revalidate: true
    
    render json: user_specific_data
  end
end

class SensitiveController < ApplicationController
  def show
    # Never cache
    response.cache_control[:no_store] = true
    response.cache_control[:no_cache] = true
    response.cache_control[:must_revalidate] = true
    
    render json: sensitive_data
  end
end

Content negotiation uses Accept headers to select appropriate response formats. Clients specify preferred media types with quality values indicating relative preference. Servers examine Accept headers to choose the best representation of a resource.

class ApiController < ApplicationController
  def show
    respond_to do |format|
      format.json { render json: resource }
      format.xml { render xml: resource }
      format.html { render :show }
    end
  end
  
  # Manual content negotiation
  def index
    accept = request.headers['Accept']
    
    if accept.include?('application/json')
      render json: resources
    elsif accept.include?('text/xml')
      render xml: resources
    else
      render html: resources
    end
  end
end

Custom headers communicate application-specific metadata. The X- prefix historically identified custom headers, though RFC 6648 deprecated this convention. Despite deprecation, X- headers remain common in practice. Custom headers enable feature flags, debugging information, rate limiting data, and request tracing.

# Custom header patterns
class ApplicationController < ActionController::Base
  before_action :add_custom_headers
  
  private
  
  def add_custom_headers
    response.headers['X-Request-ID'] = request.request_id
    response.headers['X-Runtime'] = Benchmark.realtime { yield }
    response.headers['X-Rate-Limit-Remaining'] = rate_limit_remaining
    response.headers['X-API-Version'] = API_VERSION
  end
end

# Request tracing with correlation IDs
class RequestTracer
  def initialize(app)
    @app = app
  end
  
  def call(env)
    request_id = env['HTTP_X_REQUEST_ID'] || SecureRandom.uuid
    Thread.current[:request_id] = request_id
    
    status, headers, body = @app.call(env)
    headers['X-Request-ID'] = request_id
    
    [status, headers, body]
  ensure
    Thread.current[:request_id] = nil
  end
end

ETags enable efficient caching through conditional requests. Servers generate ETags as fingerprints of response content. Clients include ETags in If-None-Match headers on subsequent requests. Servers return 304 Not Modified when content remains unchanged, saving bandwidth.

# ETag implementation
class ArticlesController < ApplicationController
  def show
    @article = Article.find(params[:id])
    
    if stale?(etag: @article, last_modified: @article.updated_at)
      render json: @article
    end
    # Rails automatically returns 304 if ETag matches
  end
  
  # Manual ETag handling
  def index
    @articles = Article.all
    etag = Digest::MD5.hexdigest(@articles.to_json)
    
    if request.headers['If-None-Match'] == etag
      head :not_modified
    else
      response.headers['ETag'] = etag
      render json: @articles
    end
  end
end

Practical Examples

API Authentication with Bearer Tokens

Modern APIs authenticate requests using tokens in Authorization headers. The Bearer scheme passes tokens without encoding, relying on HTTPS for confidentiality. Token validation occurs on every request, requiring efficient lookup mechanisms.

class ApiController < ApplicationController
  before_action :authenticate_with_token
  
  private
  
  def authenticate_with_token
    auth_header = request.headers['Authorization']
    
    unless auth_header&.start_with?('Bearer ')
      render json: { error: 'Missing authorization' }, status: :unauthorized
      return
    end
    
    token = auth_header.delete_prefix('Bearer ')
    @current_user = User.find_by(api_token: token)
    
    unless @current_user
      render json: { error: 'Invalid token' }, status: :unauthorized
    end
  end
end

Rate Limiting with Custom Headers

Rate limiting protects APIs from abuse by tracking request counts per client. Response headers communicate remaining quota and reset times, enabling clients to manage their request patterns.

class RateLimiter
  def initialize(app)
    @app = app
    @store = Redis.new
  end
  
  def call(env)
    request = Rack::Request.new(env)
    client_id = extract_client_id(request)
    
    limit = 100
    window = 3600
    key = "rate_limit:#{client_id}:#{Time.now.to_i / window}"
    
    current = @store.incr(key)
    @store.expire(key, window) if current == 1
    
    remaining = [limit - current, 0].max
    reset_time = ((Time.now.to_i / window) + 1) * window
    
    status, headers, body = @app.call(env)
    
    headers['X-RateLimit-Limit'] = limit.to_s
    headers['X-RateLimit-Remaining'] = remaining.to_s
    headers['X-RateLimit-Reset'] = reset_time.to_s
    
    if current > limit
      headers['Retry-After'] = (reset_time - Time.now.to_i).to_s
      return [429, headers, ['Rate limit exceeded']]
    end
    
    [status, headers, body]
  end
  
  private
  
  def extract_client_id(request)
    request.env['HTTP_X_API_KEY'] || request.ip
  end
end

Content Compression

Compressing response bodies reduces bandwidth and improves load times. The Accept-Encoding header indicates which compression algorithms the client supports. The Content-Encoding header identifies the compression applied to the response.

class CompressionMiddleware
  def initialize(app)
    @app = app
  end
  
  def call(env)
    status, headers, body = @app.call(env)
    
    accept_encoding = env['HTTP_ACCEPT_ENCODING'] || ''
    return [status, headers, body] unless accept_encoding.include?('gzip')
    
    # Skip compression for small or already compressed content
    content_type = headers['Content-Type']
    return [status, headers, body] if headers['Content-Encoding']
    return [status, headers, body] unless compressible?(content_type)
    
    compressed = compress_body(body)
    
    # Only use compression if it reduces size
    if compressed.bytesize < original_size(body)
      headers['Content-Encoding'] = 'gzip'
      headers['Content-Length'] = compressed.bytesize.to_s
      headers['Vary'] = 'Accept-Encoding'
      
      [status, headers, [compressed]]
    else
      [status, headers, body]
    end
  end
  
  private
  
  def compressible?(content_type)
    ['text/', 'application/json', 'application/javascript'].any? do |type|
      content_type&.start_with?(type)
    end
  end
  
  def compress_body(body)
    io = StringIO.new
    gz = Zlib::GzipWriter.new(io)
    body.each { |chunk| gz.write(chunk) }
    gz.close
    io.string
  end
  
  def original_size(body)
    body.sum { |chunk| chunk.bytesize }
  end
end

Common Pitfalls

Case Sensitivity Confusion

HTTP header names are case-insensitive per the specification, but different libraries handle this differently. Ruby's Net::HTTP normalizes header names to Title-Case, while Rack converts them to uppercase with underscores. Assuming a specific case in comparisons causes bugs.

# Wrong: case-sensitive comparison
request['content-type']  # Might not find 'Content-Type'

# Correct: case-insensitive access
request['Content-Type']
request['content-type']  # Works with proper library support

# In Rack, always use uppercase
env['HTTP_CONTENT_TYPE']  # Wrong
env['CONTENT_TYPE']       # Correct

Header Injection Vulnerabilities

Including unsanitized user input in headers enables attackers to inject malicious headers. Newline characters split headers, allowing header injection attacks. Always validate and sanitize user data before including it in headers.

# Vulnerable code
response.headers['X-Username'] = params[:username]
# Attacker sends: "admin\r\nSet-Cookie: session=stolen"

# Secure code
username = params[:username].gsub(/[\r\n]/, '')
response.headers['X-Username'] = username

# Or use framework helpers that sanitize automatically
response.headers['X-Username'] = sanitize_header_value(params[:username])

CORS Preflight Caching Issues

Browsers cache CORS preflight responses according to Access-Control-Max-Age headers. Setting this value too high causes problems when CORS configuration changes. Setting it too low increases preflight request overhead. Finding the right balance requires considering deployment frequency and performance needs.

# Too aggressive caching
response.headers['Access-Control-Max-Age'] = '86400'  # 24 hours

# Better: shorter cache for development
max_age = Rails.env.production? ? 3600 : 60
response.headers['Access-Control-Max-Age'] = max_age.to_s

Missing Vary Headers

When responses vary based on request headers, the Vary header must list those headers. Proxies and caches use Vary to store different versions of resources. Missing Vary headers cause caches to serve wrong content to clients with different capabilities.

# Wrong: content varies but Vary header missing
def show
  if request.headers['Accept'].include?('json')
    render json: data
  else
    render html: data
  end
  # Cache serves JSON to HTML clients or vice versa
end

# Correct: Vary header indicates content negotiation
def show
  response.headers['Vary'] = 'Accept'
  
  if request.headers['Accept'].include?('json')
    render json: data
  else
    render html: data
  end
end

Authorization Header Overwrite

Middleware that sets Authorization headers can overwrite existing values, breaking authentication. Defensive programming checks for existing headers before setting new values.

# Wrong: overwrites user's authentication
def call(env)
  env['HTTP_AUTHORIZATION'] = "Bearer #{default_token}"
  @app.call(env)
end

# Correct: preserves existing auth
def call(env)
  env['HTTP_AUTHORIZATION'] ||= "Bearer #{default_token}"
  @app.call(env)
end

Content-Length Mismatches

Manually setting Content-Length headers risks mismatches with actual body size. Frameworks calculate Content-Length automatically, but manual manipulation can desynchronize these values, causing truncated responses or hung connections.

# Wrong: manual Content-Length likely incorrect
response.headers['Content-Length'] = '100'
response.body = generate_body  # Actual size may differ

# Correct: let framework calculate
response.body = generate_body
# Content-Length set automatically

Reference

Standard Request Headers

Standard Response Headers

Security Headers

Cache-Control Directives

Ruby Header Access Patterns

Common Content-Type Values