Hiding Sensitive Information from Logs Using Parameter Filtering in Rails

Most applications have sensitive data they'd like to keep out of reach from prying eyes. It can be something as obvious as passwords and credit card numbers, or something less obvious but equally important, like Stripe authentication tokens.

Log files are a very common way to expose this information accidentally. Developers often like to log everything for debugging or troubleshooting purposes. Additionally, web application frameworks like Ruby on Rails log a lot of information at various stages of the request lifecycle. If you're not careful, it's very easy to leak any sensitive data in the log files.

Here's an example where the application logs out the user's password along with other information after they complete and submit the user registration form:

In this article, we will learn how to filter sensitive data from leaking into log files. Here’s a list of topics this article covers.

• Why you shouldn't log sensitive information in production
• How to filter logs in Rails with parameter filtering
• How Rails implements parameter filtering
• The ParameterFilter class API

Why you shouldn't log sensitive information

Imagine a user is making a payment on your web application. Once they enter their credit card details and hit submit, the website sends all this information in plain text to your server.

Upon receiving this request, the application logs information about this request, such as the user's IP address, user id, name, form data such as payment information, and much more.

These log files are handy when debugging the application, but in production, you may not want every bit of information to be stored in the log file.

It's very easy to accidentally leak confidential information through logs.

Here're three reasons against logging sensitive information.

1. It makes an attacker's life easy. Since most logs are stored in plain text files for readability, if your servers are compromised and an attacker gets access to the log files, they can just search the file for anything they need.
2. Data Privacy & Curious Employees. It doesn't have to be hackers. It could just be a curious employee with access to the logs who might be interested in looking into customers' sensitive data. Even if the employee may not have bad intentions, it can land you in a data-privacy hot zone for exposing user-identifiable personal information.  
3. They are leaking to a third-party app. Many applications use error tracking or performance monitoring systems such as AppSignal, New Relic, or Sentry. Whenever a request is made to your application, these tools grab all the user-submitted data and send it to their servers. This data could contain user-identifiable information.

For these reasons, you should prevent sensitive information from leaking into the log files. Here're some possible solutions to address this problem.

1. A basic, but tedious way is to review your codebase to find all the places where you're logging information and filter it to log only essential information, stripping out everything else.
2. A better solution is to make your logger check for the presence of specific sensitive attributes on the data and replace them with `***`. This is better, but it's still a lot of work.
3. The best solution is to program and abstract it via configuration. That way, you list the sensitive information in one location (the configuration file), and don't have to worry about it ever.

If you're using Ruby on Rails, you're in luck, because that's what Rails does for you via parameter filtering.

Let's understand how it works.

How to Filter Logs in Rails with Parameter Filter

Ruby on Rails makes it very easy to filter sensitive information from log files by providing a central location where you can configure this information.

For example, if you want to hide attributes named password and credit_card, add this line in the filter_parameter_logging initializer under the config/initializers directory.

# config/initializers/filter_parameter_logging.rb

config.filter_parameters += [ :password, :credit_card ]

This tells Rails to replace all instances of the password and credit_card with the word [FILTERED] and hide their real values.

Let's reconsider our previous example. This time, when the user submits the payment form, Rails will still log the user's id and name, but it will replace the credit card number with the word FILTERED and won't appear in the log files.  

Here's what the logs look like after we use parameter filtering to filter out the password and authentication tokens. Notice that the actual values are replaced with [FILTERED] mask.

Selectively Filtering Attributes with Same Name

Here's a tricky situation. Let's say you have an attribute named number for both Post and CreditCard models. The Post number is public and need not be hidden, but the Credit Card number is very sensitive information you want to hide.

Simply adding number to the filter_parameters array will hide all numbers, which is not what you want. You still want to access Post numbers for debugging.

You can solve this problem by scoping the number attribute to the CreditCard model.

config.filter_parameters += [ "credit_card.number" ]

Now Rails will only filter the credit card numbers without affecting the post numbers.

Additionally, you can pass a block (a function) to the filter_parameters array, giving you more control and flexibility while filtering. The following example retains the credit card number format while replacing the numbers with '*'.

# config/initializers/parameter_filtering.rb
 
Rails.application.config.filter_parameters += ->(key, value) { value.gsub!(/\d/, "*") if /credit_card/.match?(key) }

Filtering Attributes on ActiveRecord Models

You can use the filter_attributes method on your ActiveRecord models to tell Rails which columns should not be exposed.

class User < ApplicationRecord
  self.filter_attributes += [ :secret, :token ]
end

Now, when you inspect a user, Rails will hide the values of the above attributes:

> User.first
#=> #<id: 1, name: 'Akshay', secret: "[FILTERED]", token: "[FILTERED]">

If you're curious about how Rails implements this feature, keep reading.  

How Rails Implements Parameter Filtering

Before reading this, I encourage you to check my previous article, A Brief Introduction to Rails Initializers. Understanding how initializers work in general will help you better understand the parameter_filtering initializer.

Let's revisit our code to define sensitive attributes in the configuration. We added the following code in the filter_parameter_logging initializer under the config/initializers directory.

config.filter_parameters += [ "password" ]

The ActiveRecord::Railtie class (a fancy word for a class that initializes the gem) grabs the filter_parameters array from the config, and pushes it to the filter_attributes array.

# activerecord/lib/active_record/railtie.rb

module ActiveRecord
  class Railtie < Rails::Railtie
    initializer "active_record.set_filter_attributes" do
      ActiveSupport.on_load(:active_record) do
        self.filter_attributes += Rails.application.config.filter_parameters
      end
    end
  end
end

The filter_attributes array is defined in the ActiveRecord::Core module that gets included in the ActiveRecord::Base class, the superclass of ApplicationRecord, the class from which all your active record domain classes inherit.

module ActiveRecord
  module Core
    extend ActiveSupport::Concern
    
    included do
      self.filter_attributes = []
    end
  end
end

The ActiveRecord::Core module is a concern and adds the filter_attributes array on the ActiveRecord::Base class.

Check out this article to learn more about concerns and how they work.

Concerns in Rails: Everything You Need to Know

Concerns are an important concept in Rails that can be confusing to understand for those new to Rails as well as seasoned practitioners. This post explains why we need concerns, how they work, and how to use them to simplify your code.

Akshay's BlogAkshay Khot

Now let's inspect how Rails uses the filter_attributes array for filtering sensitive information.

All the methods dealing with this array are included under the ClassMethods module. Hence they become class methods on the ActiveRecord::Base class.

module ActiveRecord
  module Core
    extend ActiveSupport::Concern
    
    module ClassMethods
      def filter_attributes 
      end

      def filter_attributes=(filter_attributes)
      end

      def inspection_filter
      end
    end
  end
end

Step 1: Set the filter_attributes. Here, we initialize the class instance variable @filter_attributes.

def filter_attributes=(filter_attributes)
  @inspection_filter = nil
  @filter_attributes = filter_attributes
end

Step 2: The inspection_filter method creates an instance of the ActiveSupport::ParameterFilter class, which is responsible for filtering the parameters. The mask is set to [FILTERED] by default.

Here's the simplified implementation of this method.

def inspection_filter
  @inspection_filter ||= begin
    mask = InspectionMask.new(ActiveSupport::ParameterFilter::FILTERED)
    ActiveSupport::ParameterFilter.new(@filter_attributes, mask: mask)
  end
end

Whenever Rails needs to filter sensitive information, it uses the inspection_filter method as follows.

There're only three places in the entire Rails codebase where inspection_filter is used.

Here's the highly simplified version of these methods.

# activerecord/lib/active_record/core.rb
def pretty_print(pp)
  value = inspection_filter.filter_param(attr_name, value)
end

# activerecord/lib/active_record/log_subscriber.rb
def filter(name, value)
  ActiveRecord::Base.inspection_filter.filter_param(name, value)
end

# activerecord/lib/active_record/attribute_methods.rb
def format_for_inspect(name, value)
  # remaning code
  inspection_filter.filter_param(name, value)
end

Whenever you try to print, log, or inspect an ActiveRecord model, Rails will use the inspection_filter to hide the value of the sensitive attributes.

So far, we've been looking at the infrastructure code, i.e., how Rails defines and uses the filtered array of attributes. Now let's try to understand the core of the solution, the ActiveSupport::ParameterFilter class.

How ParameterFilter Class Works

This class replaces the values for specified keys from a hash-like object.

Initialize an instance of this class with an array of keys and then call either filter(params) or filter_param(key, value) on this instance. These methods replace the sensitive values with a mask, which is [FILTERED] by default.

• The filter(params) method replaces all values corresponding to filters with the mask
• The filter_param(key, value) replaces the value with a mask if the key belongs to the filters list, which is provided during the initialization.

Here's the external API of the ParameterFilter class.

module ActiveSupport
  class ParameterFilter
    # default mask
    FILTERED = "[FILTERED]"
    
    def initialize(filters = [], mask: FILTERED)
    end
    
    # replace all values corresponding to filters with the mask
    def filter(params)
    end
    
    # replace the value with mask if key belongs to filters 
    def filter_param(key, value)
    end
  end
end

The following test shows how it works:

test "parameter filter" do
  parameter_filter = ActiveSupport::ParameterFilter.new(['foo'])
    
  assert_equal ({ "foo" => "[FILTERED]", "bar" => "baz" }), parameter_filter.filter({ "foo" => "bar", "bar" => "baz" })
  assert_equal "[FILTERED]", parameter_filter.filter_param("foo", "bar")
end

You might be surprised to find that ParameterFilter class can also accept a block/lambda as one of the filters. During replacement, this block will be called with the provided key-value pair, and you can replace the value in place.

This example replaces the credit card numbers while preserving their format.

test "parameter filter with block" do
  filter_params = []
  filter_params << ->(key, value) { value.gsub!(/\d/, "*") if /credit_card/.match?(key) }
    
  parameter_filter = ActiveSupport::ParameterFilter.new(filter_params)

  assert_equal ({ "credit_card" => "**** ****" }), parameter_filter.filter({ "credit_card" => "9999 9999" })
  assert_equal "*****", parameter_filter.filter_param("credit_card", "12345")
end

Internally, the ParameterFilter class uses the CompiledFilter class to filter the values with the mask. It accepts the filters and a mask to build a list of regular expressions that replace the values corresponding to filters with the provided mask.  

And that's how Rails implements the parameter filtering feature to hide sensitive information. If you haven't used it, give it a try!

Summary

• You should prevent sensitive, user-identifiable information from leaking into the log files. The log files are typically stored in plain text, and it's very easy to reveal confidential data accidentally.
• The parameter filtering feature in Rails allows you to filter important information from the logs.

I hope you found this article useful and that you learned something new.

If you have any questions or feedback, please send me an email. I look forward to hearing from you.

Please subscribe to my blog below if you'd like to receive future articles directly in your email. If you're already a subscriber, thank you!