BTRFS-ism: Metadata “Evolution” – Your Backup Drive is Dead!

TL;DR if you are looking for a quick solution check BTRFS-ism: The “Corrupt Leaf” Mystery!

The Scenario

If you didn’t know what BTRFS-ism is: I found myself stuck today when I plugged in my external USB backup drive:

$ sudo mount /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup
mount: /media/backup: can't read superblock on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8.

LUKS was opened successfully, but BTRFS refused to mount. Yes, it is BTRFS – again :(

AI to the rescue?!

Gemini didn’t tell me about, but it’s a good advice to go along by tailing to system messages when dealing with FS/IO related operations:

$ dmesg -wH

Try Mounting in “Recovery” Mode

It’s obvious that AI suggested that first

$ sudo mount -t btrfs -o recovery,ro /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup
mount: /media/backup: wrong fs type, bad option, bad superblock on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8, missing codepage or helper program, or other error.

And DMESG said following which probably would not have been noticed:

btrfs: Unknown parameter 'recovery'

But also no luck with the correct options (see later section again)

Use the BTRFS Check Tool

$ sudo btrfs check /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8
UUID: c5266ee8-afde-4c0f-b50f-b4feae7c1df1
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups
found 2942485991424 bytes used, no error found
total csum bytes: 2869685492
total tree bytes: 3928047616
total fs tree bytes: 683933696
total extent tree bytes: 68075520
btree space waste bytes: 376367783
file data blocks allocated: 2938557943808
 referenced 2977425936384

Even if dmesg already reported following AI there is not yet a hint by Gemini to look out for that:

[  +0.000002] BTRFS critical (device dm-2): corrupt leaf: block=830717952 slot=86 extent bytenr=2327262031872 len=36864 invalid data ref objectid value 18446744073709551604
[  +0.000004] BTRFS error (device dm-2): read time tree block corruption detected on logical 830717952 mirror 2
[  +0.000013] BTRFS error (device dm-2): failed to read block groups: -5
[  +0.000749] BTRFS error (device dm-2): open_ctree failed: -5

But it least Gemini it tells:

Btrfs is powerful but sensitive.

Do not run btrfs check --repair unless you have no other choice; it can sometimes make corruption worse if the underlying issue is hardware-related.

Check the “Super-Recover” tool

After telling Gemini the results it continues with other checks:

Btrfs stores superblocks at fixed locations ($64\text{ KiB}$, $64\text{ MiB}$, $256\text{ GiB}$). If the primary one is slightly shifted or unreadable by the kernel (even if btrfs-progs can see it), we can verify the backups.

$ sudo btrfs rescue super-recover /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8
All supers are valid, no need to recover

The “Dirty Log” (The Nuclear Option)

Wiping replay logs. Good choice. Usually only clears last changes to files. Low risk involved here (but who knows?).

Since btrfs check passed (meaning your data is there), the log tree is likely the only thing blocking the kernel. We should try to wipe the log tree entirely. This will discard the last few seconds of writes from before the last crash/unmount, but it usually allows the drive to mount.

$ sudo btrfs rescue zero-log /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8
Clearing log on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8, previous log_root 0, level 0

Check for a Sector Size / Page Size Mismatch

Now Gemini extends the context and continues with other checks

If you recently moved this drive between different computers (e.g., from an ARM Mac/Raspberry Pi to an Intel PC), you might have a Page Size mismatch ($4\text{K}$ vs $16\text{K}$ or $64\text{K}$).

Good catch, but I do think that’s misleading and AI is missing information. External drive is an HDD, thus it does not share NVMe block sized. Nope. This is not an error, but like it should be.

$ sudo btrfs inspect-internal dump-super /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 | grep nodesize
nodesize		16384
$ getconf PAGE_SIZE
4096

Try Mounting in “Recovery” Mode

Now after issuing commands Gemini finally pointed out DMESG and the correct mount options – anyway, still no luck.

$ sudo mount -o defaults,ro,degraded,nospace_cache /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup     
mount: /media/backup: can't read superblock on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8.

Check for “Metadata Features” Mismatch

Gemini is again extending the picture now

If this drive was formatted on a newer Linux distribution (or a different architecture), it might have features enabled that your current kernel doesn’t support.

Look for: block_group_tree or extent_tree_v2. If you see these and you are on an older kernel (like an LTS kernel or an older server), the kernel will report a “bad superblock” because it doesn’t know how to handle those features in Read-Write mode.

$ sudo btrfs inspect-internal dump-super /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 | grep compat_ro_flags
compat_ro_flags		0x0

The “Ghost Mount” Check

Sometimes Btrfs gets confused if it thinks the device is already part of a multi-device pool or if the UUID is partially registered elsewhere.

Good hint, but not sure if that would help :/

$ sudo btrfs device scan --forget
$ sudo btrfs device scan

Let’s extend by:

$ command lsblk -noUUID | grep -v '^$' | sort | uniq -c
      1 5c3931a2-4333-415f-8e58-b3b4d728f4d8
      1 83cc81ff-0d0a-471c-802e-5b51d00f7e79
      1 aa96566d-75c6-46a9-9a65-4933ccae6d70
      1 c5266ee8-afde-4c0f-b50f-b4feae7c1df1
      1 e3dced76-50b1-4407-b90a-828dff61a12a
      1 f6ac36b8-aa21-4bf3-b4c0-275d3f05c306

Nope. No duplicates.

Last Resort: Emergency Restore – it works!

Since the kernel-level driver is the bottleneck here, your most effective tools are the userspace tools that don’t care about the kernel’s page size limitations:

btrfs-find-root: If you want to see where the backup roots are. sudo btrfs-find-root /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8

btrfs restore: This is your “get out of jail free” card. It bypasses the mount logic entirely. sudo btrfs restore -v /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /path/to/another/drive

$ btrfs-find-root /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8
Superblock thinks the generation is 2030
Superblock thinks the level is 1
Found tree root at 1020794339328 gen 2030 level 1
Well block 1020794060800(gen: 2029 level: 1) seems good, but generation/level doesn't match, want gen: 2030 level: 1
Well block 1020793716736(gen: 2028 level: 1) seems good, but generation/level doesn't match, want gen: 2030 level: 1
Well block 1020792782848(gen: 2027 level: 1) seems good, but generation/level doesn't match, want gen: 2030 level: 1
Well block 1020792225792(gen: 2026 level: 1) seems good, but generation/level doesn't match, want gen: 2030 level: 1
…
Well block 1020533538816(gen: 403 level: 0) seems good, but generation/level doesn't match, want gen: 2030 level: 1
Well block 1020533522432(gen: 403 level: 0) seems good, but generation/level doesn't match, want gen: 2030 level: 1
Well block 1020533506048(gen: 403 level: 0) seems good, but generation/level doesn't match, want gen: 2030 level: 1
Well block 1020533440512(gen: 403 level: 0) seems good, but generation/level doesn't match, want gen: 2030 level: 1

Gemini said

This is great news! btrfs-find-root just showed you that your disk is a history book of previous “known-good” states.

Because Btrfs is Copy-on-Write (CoW), it doesn’t overwrite old metadata; it writes new versions elsewhere. Your current generation (2030) contains that “corrupt leaf” that the new kernel hates. But generation 2029, 2028, etc., are still sitting there on the platters, likely perfectly intact.

Since your kernel is 6.12, you can use the modern rescue syntax to tell the kernel to “ignore the current broken state and go back to a previous generation.”

Try Mounting a Previous Generation

We will tell the kernel to use the backup roots. It will automatically try to find the most recent valid generation before the one that failed.

Run this command: sudo mount -t btrfs -o ro,rescue=usebackuproot /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup

Why this works: It ignores the “Tree Root” pointed to by the superblock (gen 2030) and looks for the backup tree roots (gen 2029, 2028, etc.).

Nope it didn’t work. Same error.

If the kernel is still too “smart” (or strict) and refuses to mount even the older generations, you can use the restore tool and manually point it to one of those addresses found by btrfs-find-root.

$ sudo btrfs restore -t 1020794060800 -v /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 ~/recovery_destination

But also it seems to work with old root.

$  sudo btrfs restore -v /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 ~/recovery_destination

Anyway, stop here! This is not the end. We can do better!

Human Intervention

Let’s tell Gemini a bit more about the setup

/me

It’s an older device created by kernel 5.14 and space_cache=v1. Now I am running kernel 6.12.59-x64v3

Gemini

That explains the friction perfectly. You are moving from a very old “stable” Btrfs state (v1 cache, older kernel) into a modern environment that has much stricter validation rules for metadata.

The BTRFS critical: corrupt leaf error you saw isn’t necessarily a physical disk failure; it’s often a “logical” corruption that occurs when newer kernels (like your 6.12.59) perform more rigorous tree-integrity checks than the kernel that wrote the data.

$ sudo mount -t btrfs -o ro,rescue=nologreplay,clear_cache,space_cache=v2 \
  /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 \
  /media/backup
mount: /media/backup: can't read superblock on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8.

space_cache=v2 nope. And still the Superblock problem, so makes no sense beating dead horses with mount options.

If you suspect there is a specific secific tree error blocking the mount (and not just the page size issue), Btrfs uses the usebackuproot option instead of a superblock flag. This tells the kernel to use the backup “tree roots” (the pointers to your data):

Note: On older kernels, use -o recovery instead of -o rescue=usebackuproot.

$ sudo mount -o defaults,ro,rescue=usebackuproot,degraded,nospace_cache,clear_cache,subvol=@ /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup
mount: /media/backup: can't read superblock on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8.

We’re pasting the DMESG log again and Gemini finally

This confirms the “why” behind the failure. The kernel is attempting to read your Block Groups (the metadata that tells Btrfs how space is allocated), but it hits a corruption in a metadata “leaf.”

Specifically, the value 18446744073709551604 is a very large number that the modern kernel (6.12) considers an invalid object ID. Because this happens while reading block groups, the mount fails with error -5 (EIO - Input/output error).

Step 1: Use btrfs restore to grab your most important files. Step 2: Once your data is safe, try the btrfs check --repair --init-extent-tree repair.

Nope. I’m not satisified here with the solution…

Human Intelligence – and mounting works on an older system!

We’re attaching the LUKS mapped device to a KVM instance:

$ aria2c -j8 https://releases.ubuntu.com/focal/ubuntu-20.04.6-live-server-amd64.iso
$ sudo qemu-system-x86_64 \
  -enable-kvm \
  -m 2G \
  -cpu host \
  -cdrom ubuntu-20.04.6-live-server-amd64.iso \
  -drive file=/tmp/cloud-init-RLrcc/seed.iso,format=raw,media=cdrom \
  -drive file=/dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8,format=raw,if=virtio \
  -boot d

Press F2and once in the VM’s root shell:

$ mkdir /mnt/backup
$ mount /dev/vda /mnt/backup
$ ls /mnt/backup
…

…and this works?! YES!!! But also not with my other Laptop running an up-to-date Fedora: same error. And we’re still not at the end.

The Real Problem: Metadata Feature Mismatch

After telling our AI about our A/B testing Gemini concludes:

That specific error message is a “smoking gun.” The value 18446744073709551604 is actually a very large unsigned integer representing -12 in two’s complement.

When you see a Btrfs error involving an invalid objectid of -12, it almost always points to a feature called extent-tree-v2 or block-group-tree.

Your older system (Ubuntu 20) is likely running a kernel/progs combo that either doesn’t check for this specific metadata consistency or has a different way of handling 64-bit extent references.

Your newer system (Kernel 6.15) has much stricter tree-checker logic. It sees a “corrupt leaf” because the metadata written by the older tools/kernel doesn’t conform to the strict validation rules of the 6.15 kernel.

Why this happens with “New Kernel + Old Progs”

When you create a Btrfs filesystem with older btrfs-progs on a newer kernel:

1. The Kernel wants to use modern defaults (like no-holes or v2 space cache).
2. The Progs (v5.16) might not correctly initialize the metadata headers or the “features” flags in the superblock for those defaults.
3. The Result: You end up with a “hybrid” filesystem that is technically valid according to old standards but triggers “corrupt leaf” warnings on new, stricter kernels.

At least we are now able to mount read-only via rescue=all and skipping all tests!

At least don’t need to run btrfs restore and can directly access an BTRFS FS or snapshots

 sudo mount -o rescue=all,ro /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup            

DMESG

[  +0.017638] BTRFS info (device dm-2 state CS): disabling log replay at mount time
[  +0.000004] BTRFS info (device dm-2 state CS): enabling disk space caching
[  +0.000001] BTRFS info (device dm-2 state CS): ignoring bad roots
[  +0.000001] BTRFS info (device dm-2 state CS): ignoring data csums
[  +0.000001] BTRFS info (device dm-2 state CS): ignoring meta csums
[  +0.000000] BTRFS info (device dm-2 state CS): ignoring unknown super block flags

But don’t think you could now mount with r/w!

By using rescue=all, you are telling the kernel: “Ignore the Tree-Checker.” The kernel will stop complaining about the invalid data ref objectid and allow the mount. However, because you are mounting in rw (read-write) mode, the kernel will attempt to update the metadata. If it writes new data based on a “leaf” it already thinks is weird, it could technically propagate that “weirdness” to other parts of the tree.

Your old photos/files are likely safe because Btrfs won’t overwrite them (due to CoW).

This is where the real danger lies. You risk Metadata Corruption. If the tree structure becomes sufficiently scrambled because the kernel was “ignoring” errors while writing, the drive might eventually become unmountable even on your “good” Ubuntu 20.04 system.

OH RLY…?! My “old photos/files are likely safe”?! Just that I cannot access them anymore!!! :roll_eyes:

That’s also what data brokers say, but they won’t get insight in what they are really storing.

Breaking down on incompatible features: incompat_flags=0x371

So, can we switch off the incompatible flags or instead of repairing of what is not broken.

The term incompat_flags (Incompatible Flags) sounds scary, like something is broken or “incompatible” with your hardware. In the Linux kernel world, however, it means something very specific: This filesystem uses features that are incompatible with OLDER versions of Linux.

$ sudo btrfs inspect-internal dump-super /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 | grep incompat_flags -A 5
incompat_flags		0x371
         	( MIXED_BACKREF |
			  COMPRESS_ZSTD |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA |
			  NO_HOLES )

In Btrfs, 0x371 translates to the following active features – please verify!

BTW Gemini spit out RAID56 by only giving the bitmask – which was wrong!

Your error (invalid data ref objectid ...) combined with these flags points to a specific collision between SKINNY_METADATA and NO_HOLES.

On older kernels, these features were handled more “loosely.” On Kernel 6.15, the Tree-Checker is extremely aggressive. If the older btrfs-progs (5.16) created the metadata leaves but didn’t align the data references perfectly with the “Skinny” format expectations, the new kernel sees that “Global Block Group” ID (-12) and assumes the leaf is corrupted to prevent a system crash.

Since you cannot easily use btrfstune to strip these flags in version 5.16, you must use the Ubuntu 20.04 system (the one that still trusts the drive) to “normalize” the metadata.

BTRFS’ism: Nothing is broken – you just cannot use it anymore!?

I should close with these words …

That phrase is a classic “Btrfs-ism.” It usually describes a state where the filesystem is technically consistent (not corrupted), but it has entered a Read-Only (RO) safety mode because it can no longer guarantee safe writes.

Next is to buy a new drive or find a space to save everything to, wait for severals hours to finish just to create… a new BTRFS-filesystem!???

It made a great leap from space_cache=v1 to v2 and newer features and got now pretty mature from Kernel 5.x to 6.x.

Final result: Don’t just unplug! Unmount and rebalance first!

The balance option under the the old stable-system should still be safe. So, let’s boot up Ubuntu 20 again and do r

 $ mount /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /mnt/backup        

And once we do the re-belancing and duplication of the metadata don’t forget to do with --bg (run i nbackground) so we can watch the progress ;)

$ btrfs balance status -mconvert=dup --bg /mnt/backup
Done, had to relocated 13 out of 2816 chunks
$ watch -n1 btrfs balance status /mnt/backup

With a 5TB HDD volume attached via USB3.0 this now took 5mins.

Finally, Mounting the Volume is Working Again!

Back on our newer system we’re finally able to mount the drive again:

mount /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup        

Lesson Learned: when you’re using a BTRFS under an older system (or a BTRFS module with older features), don’t just unplug.

Sorry, I’m still not jumping on the OpenZFS or bcachefs band-waggon.

**UPDATE: Still not sure if I should hate it now … **

Yes, I thought it would work with r/w again…

$ mkdir -p /media/backup/snapshots/thiccbook

DMESG

[  +0.000000] BTRFS error (device dm-2): block=6363830763520 write time tree block corruption detected
[  +0.010717] BTRFS: error (device dm-2) in btrfs_commit_transaction:2540: errno=-5 IO failure (Error while writing out transaction)
[  +0.000003] BTRFS info (device dm-2 state E): forced readonly
[  +0.000005] BTRFS warning (device dm-2 state E): Skipping commit of aborted transaction.
[  +0.000001] BTRFS error (device dm-2 state EA): Transaction aborted (error -5)
[  +0.000001] BTRFS: error (device dm-2 state EA) in cleanup_transaction:2027: errno=-5 IO failure

I hope it’s not just a rumour that BTRFS has now reached maturity with Linux 6.x. Still, I’m opposed to OpenZFS and bcachefs is not yet an alternative :pensive:

Always keep a backup and decent snapshots when working with BTRFS.