Shooting both feet with BTRFS replace and NVME OPAL encryption

TLDR: Don’t run btrfs replace unless you fully understand its implications! (Worry not, there is a solution if you already made a mistake)

Intro

Recently I decided to move to a new PC and in the process also move my Linux installation to a new bigger NVME drive.

Most (all?) SSD/NVME drives already internally encrypt the data (such disks are called self-encrypting drives (SED)). There are certain benefits of such encryption even when it stores the key in plain text and doesn’t require user password, e.g. fast full-disk wipe. Also, some drives implement OPAL specification and can encrypt the internal encryption key with user supplied key, thus providing full-disk encryption for free.

Even if a manufacturer states that a drive supports OPAL it doesn’t mean it would work in practice or is secure.

I already used this feature on another installation, but on the current installation I wanted to move I used software disk encryption, which isn’t fast enough. So I decided to enable built-in NVME encryption on my new drive, then copy BTRFS filesystem via btrfs replace which helpfully stated:

On a live filesystem, duplicate the data to the target device which is currently stored on the source device. If the source device is not available anymore, or if the -r option is set, the data is built only using the RAID redundancy mechanisms. After completion of the operation, the source device is removed from the filesystem.

What could go wrong? “After completion of the operation, the source device is removed from the filesystem.” - removed is not “deleted” or “wiped”, right? No warnings in sight.

Attempt to setup encryption and copy data

• Set up encryption for NVME via sedutils, added PBA image to decrypt it. See Self-encrypting_drives on Arch wiki for details.
  • Note, Pre-boot Authorisation step could be avoided by e.g. using mkinitcpio hook, also instead of sedutils one could use cryptsetup-luksFormat, which in theory may be more robust and maintainable, since sedutils are not well maintained.
• Tested that drive could be unlocked with the password. But did NOT test that it could be unlocked via PBA at boot time.
• Ran btrfs replace from the old to the new drive.
• Tried to boot into the new drive.
• NVME cannot be unlocked…
• Ok, whatever, I could just remove the encryption since I know the password, right?
• Booting previous NVME drive I copied the system from… kernel cannot mount the partition.
• Carefully looked at LUKs encryption there, it descrypts the partition, but it doesn’t get recognized as any filesystem.
• Booted in Windows, trying to unlock the new drive - NOT_AUTHORIZED is what sedutils tells me???
• Booted in Linux live USB - sedutils cheerfully tells me NOT_AUTHORIZED…
• Repeated both steps and iterated on potential ways I could have mistyped password.
• At this step I have old drive with non-existing filesystem and new drive which cannot be decrypted. That was rather unexpected. And while I have a backup, it’s a week old and I’d like to avoid restoring from it.
• Ok, let’s get back to old drive and understand what happened with it.
• After searching a bit I found that btrfs replace helpfully WIPES MAGIC BTRFS STRINGS FROM THE FILESYSTEM. Probably to teach us a lesson of not running commands unless we understand every single word of there description, I don’t know.
• The good news is that the command is lazy and wipes ONLY the magic strings.

The solution for btrfs replace woes

Fortunately I was not the only one doing this mistake and found a single answer to this issue: https://superuser.com/questions/1281942/recover-data-from-replaced-btrfs-disk/1715352#1715352, which I will duplicate here:

# backup magic strings just in case
dd bs=1 count=8 if=/dev/sdb skip=$((64*1024+64)) of=~/magic1
dd bs=1 count=8 if=/dev/sdb skip=$((64*1024*1024+64)) of=~/magic2
dd bs=1 count=8 if=/dev/sdb skip=$((256*1024*1024*1024+64)) of=~/magic3

# write magic string
echo "_BHRfS_M" | dd bs=1 count=8 of=/dev/sdb seek=$((64*1024+64))
echo "_BHRfS_M" | dd bs=1 count=8 of=/dev/sdb seek=$((64*1024*1024+64))
echo "_BHRfS_M" | dd bs=1 count=8 of=/dev/sdb seek=$((256*1024*1024*1024+64))

btrfs device scan
mkdir -p /mnt/before_replace
mount -o degraded,ro /dev/sdb /mnt/before_replace

And it absolutely worked! I had to initially mount BTRFS in degraded mode, but I think btrfs check is what helped it.

After booting into old disk I was able to disable encryption of the new disk without any trouble. I still don’t know why descryption worked only from that particular installation…