💎Migrate attr_encrypted to Rails 7 Active Record encrypts

In this guide, I’ll show you how to migrate away from attr_encrypted to the new Active Record encrypts.

Rails 7 has introduced Active Record Encryption, functionality to transparently encrypt and decrypt data before storing it in the database. This is awesome news for any developer who has ever had to encrypt data before storing it.

In this guide, I will walk you through an example to migrate from away from using attr_encrypted gem to the new Rails 7 Active Record encrypts. We will do this using strong migrations and also maintain the ability to perform a database rollback without data loss.

TLDR

If you are short on time, below is the crux of this article. If you are actually implementing this, I would highly encourage you to read on, this can be a fairly complex migration.

Important notes

• This article is written on 13 April 2021 - Currently Rails 7 is Edge (aka alpha). This tutorial makes certain assumptions based on that. I will publish updates when Rails 7 is officially released.

• attr_encrypted and Active Record encrypts are not compatible - you’ll need to use a fork of attr_encrypted

• devise - currently needs the patch-2 branch to work with Rails 7

Basic Process

1. Upgrade to Rails 7

2. Add dynamic attributes to model

3. Perform Migrations

4. Delete attr_encrypted gem dependency

Background

Most applications at some point in time need to encrypt data before storing it to the database (and conversely decrypt it before using it in the application). Historically there have been 2 gems that were fairly popular for this sort of functionality, namely attr_encrypted and lockbox. I personally have preferred lockbox since its still actively maintained and used less columns, but if you are like me you can’t always choose whats handed to you.

Unfortunately, the attr_encrypted gem is no longer maintained has a lot of name clashes with the Rails 7 Active Record encrypts functionality. To work around this, we had to create a fork and rename many of the function calls and properties (namely encrypt, decrypt, ect.). You too will need to use the PagerTree fork of the attr_encrypted gem during your migration process (but don’t worry, you can delete it after your migration).

Upgrade to Rails 7

You’ll first need to upgrade to Rails 7. As of this writing (13 April 2021), Rails 7 is Edge. This tutorial will use syntax and functionality that is currently in alpha.

In your gem file you’ll need to change:

Gemfile

# Use edge rails, currently 7.0
gem 'rails', github: 'rails/rails'
# Until this is officially release in devise, we need to use the fixed branch
gem "devise", github: "ghiculescu/devise", branch: 'patch-2'
# attr_encrypted is no longer maintained. There are lots of function names that clash with 
# Rails 7 encrypts functionality. This branch has renamed the conflicting functions 
# you will be able to remove this after the migration
gem "attr_encrypted", github: "PagerTree/attr_encrypted", branch: "rails-7-0-support"

And then make sure to install the new dependencies, and update any others.

bundle install
bundle update

At this point, we now have Rails 7 installed, with a compatible version of attr_encrypted.

Add Rails 7 Active Record encrypts keys

Following encrypts documentation, you’ll need to add some keys to your rails credentials file.

bin/rails db:encryption:init

Copy the output YAML and paste it into your credentials file. It should look something like this:

active_record_encryption:
  primary_key: EGY8WhulUOXixybod7ZWwMIL68R9o5kC
  deterministic_key: aPA5XyALhf75NNnMzaspW7akTfZp0lPY
  key_derivation_salt: xEY0dt6TZcAMg52K7O84wYzkjvbA62Hz

You’ll need to do this once for each environment (normally development, staging, production).

rails credentials:edit --environment=development

At this point, the Active Record encrypts should be ready to go.

Migrate attr_encrypted

The next steps will be to migrate any data that was previously using attr_encrypted to use the new encrypts methods. Because we want to be secure and also use strong migrations our process should look like this:

1. Modify our model to dynamically define attributes to use during our migration

2. Create a new temporary column for our old encrypted data (attr_encrypted)

3. Copy old encrypted data to new temporary column, and delete old column

4. Add a new column for our Rails 7 Active Record encrypts data

5. Run a migration to programmatically decrypt attr_encrypted temporary column and put it in Rails 7 Active Record encrypts column

6. Delete temporary column

Seems like a lot of overkill, but we do it this way so we don’t perform any dangerous database activity and make strong migrations happy. This process will also keep our migrations backward compatible and prevent data loss in case we ever need to rollback.

Assumption

I’m going to make the assumption you are fairly dangerous when it comes to coding, and you are relatively familiar with the rails framework. Please use this example as a guide. You’ll need to make modifications to your own code to make this work for you..

Below, is what I will assume is our starting point. We have a User model that has an attribute called otp_secret (stands for “one time password secret”, for two factor authentication).

class User < ApplicationRecord
    ...
    attr_encrypted :otp_secret, key: Base64.decode64(Rails.application.credentials.otp_secret_encryption_key)
end

The otp_secret property currently uses attr_encrypted. This means in our database we should have the following columns:

column_name	column_type
encrypted_otp_secret	:string
encrypted_otp_secret_iv	:string

We’ll take advantage of the fact that attr_encrypted prefixed its column names with “encrypted”. By copying our data into a temporary column, we can avoid name clashes, and use the encrypts functionality almost transparently (you’ll see below how the names will come full circle).

Modify model to use dynamic attributes

We need to add some extra code to dynamically define attributes. During the migration, only two of these columns will ever exist at a time, making it so that we can migrate our columns without name clashing.

class User < ApplicationRecord
...
  if column_names.include? "encrypted_otp_secret"
    attr_encrypted :otp_secret, key: Base64.decode64(Rails.application.credentials.otp_secret_encryption_key)
  end

  if column_names.include? "encrypted_otp_secret_2"
    attr_encrypted :otp_secret_2, key: Base64.decode64(Rails.application.credentials.otp_secret_encryption_key)
  end

  if column_names.include? "otp_secret"
    encrypts :otp_secret 
  end
end

Create a Temporary Column

The temporary column will just hold a copy of our existing attr_encrypted field. We move data here for strong migrations and so the Rails 7 encrypts column doesn’t conflict with the attr_encrypted accessor.

rails g migration AddEncryptedColumnToUsers

class AddEncryptedColumnToUsers < ActiveRecord::Migration[7.0]
  def change
    add_column :users, :encrypted_otp_secret_2, :string
    add_column :users, :encrypted_otp_secret_2_iv, :string
  end
end

Copy attr_encrypted columns

You’ll want to create a new migration that copies the original attr_encrypted column to the one we just created, but you’ll want to make sure you define both up and down so that you can have backward compatibility.

rails g migration CopyEncryptedColumnsOnUsers

class CopyEncryptedColumnsOnUsers < ActiveRecord::Migration[7.0]
  def up
    User.update_all("encrypted_otp_secret_2=encrypted_otp_secret")
    User.update_all("encrypted_otp_secret_2_iv=encrypted_otp_secret_iv")
    safety_assured {
      remove_column :users, :encrypted_otp_secret
      remove_column :users, :encrypted_otp_secret_iv
    }
  end

  def down
    add_column :users, :encrypted_otp_secret, :string
    add_column :users, :encrypted_otp_secret_iv, :string
    User.update_all("encrypted_otp_secret=encrypted_otp_secret_2")
    User.update_all("encrypted_otp_secret_iv=encrypted_otp_secret_2_iv")
  end
end

Add Rails 7 Active Record encrypts columns

Now we’ll add a new column, where we will store the Rails 7 Active Record encrypts data.

It’s important that the column be of type :text. The rails guides specify that the column should be at least 510 bytes.

rails g migration AddOtpSecretToUsers

class AddOtpSecretToUsers < ActiveRecord::Migration[7.0]
  def change
    add_column :users, :otp_secret, :text
  end
end

This takes full advantage of the previous attr_encrypted nomenclature. After the migration, we should be able to access the opt_secret still by using user.otp_secret. See how that just came full circle 😉

Migration data from attr_encrypted to encrypts

In this step, we generate a migration to move data from the attr_encrypted property to the Rails 7 Active Record encrypts property. We have to do this programmatically (and can’t do a shortcut db command) because it is the rails engine is what is actually doing the encrypt and decrypt work for us.

Additionally, we do some special reloading of the User model because of how we have dynamically defined attributes (Again, this is meant just to be temporary while we migrate).

rails g migration PortAttrEncryptedToEncrypts

class PortAttrEncryptedToEncrypts < ActiveRecord::Migration[7.0]
  def up
    reload_user_model
    User.all.each do |u|
      # takes the attr_encrypted properties and puts in the Rails 7 properties
      # must do this programmatically because thats how encryption happens.
      # We can't shortcut this via a db command
      u.otp_secret = u.otp_secret_2
      u.save!
    end
    reload_users_model
  end

  def down
    reload_users_model
    User.all.each do |u|
      # takes the Rails 7 properties and puts in the attr_encrypted properties
      # must do this programmatically because thats how encryption happens.
      # We can't shortcut this via a db command
      u.otp_secret_2 = u.otp_secret
      u.save!
    end
    reload_users_model
  end

  def reload_users_model
    # We must do this because of how the User model is
    # dynamically defined
    User.reset_column_information
    Object.send(:remove_const, "User")
    load "app/models/user.rb"
  end
end

Remove Temporary Column

Our last step is to remove our temporary column, so our database is kept nice and clean. Again, we define the up and down methods in this migration so we are backward compatible, and if for any reason we can go back in time and re-create our data.

rails g migration RemoveAttrEncryptedColumnsFromUsers

class RemoveAttrEncryptedColumnsFromUsers < ActiveRecord::Migration[7.0]
  def up
    safety_assured {
      remove_column :users, :encrypted_otp_secret_2
      remove_column :users, :encrypted_otp_secret_2_iv
    }
  end

  def down
    add_column :users, :encrypted_otp_secret_2, :string
    add_column :users, :encrypted_otp_secret_2_iv, :string
  end
end

Run your migrations

Now you should be able to run all your newly created migrations with one swift command.

rails db:migrate

If it worked, congrats 😄 If for some reason it doesn’t work, check the error output. It could be a simple syntax error, or something specific to your setup. Here is where I am counting on you to be dangerous and figure out what could have happened.

Remove attr_encrypted dependancy

You can now safely remove the attr_encrypted dependancy in your gem file. However, be aware that this will break existing process of rails db:create db:setup (for example in development). You’ll likely want to use rails db:setup instead so that it loads from the schema file and at some point squash your migrations directory.

---

I hope you find some value in this tutorial and it can save you time and effort when it comes to migrating away from attr_encrypted. There’s probably a lot I missed on here, so if you have something to add you can reach out to me on twitter and I will update the article with your suggestion.

---

Other Notes

Some other notes on snags I came across during development.

Problems when creating a database with Devise and Rails 7 Active Record Encrypts

The Rails 7 Active record encrypts seems to break db:create when used in conjunction with Devise. Didn’t dig too far into this, but Rails complains that the encrypts modifier can’t properly check the database column size. Makes sense since there currently is no database, but it did force me to create a hack on the user model. It didn’t seem to affect other models that didn’t interact with Devise.

I assume this will get fixed at some point and is just a Devise + Edge (alpha) thing.

# Only declare encrypts attributes when we have a database
if (::ActiveRecord::Base.connection_pool.with_connection(&:active?) rescue false)
  encrypts :otp_secret
end