BTRFS-ism: The “Corrupt Leaf” Mystery – or when newer kernels turn strict
TL;DR if you are into a longer version including how to debug check BTRFS-ism: Metadata “Evolution” – the story so far!
If you’ve moved a BTRFS drive from an older system (like Ubuntu 20.04) to a cutting-edge kernel (like 6.15) and suddenly hit a “can’t read superblock” on mounting your volume, you aren’t alone. Even weirder: the drive still works perfectly on the old system.
$ sudo mount /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup
mount: /media/backup: can't read superblock on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8.
It’s not clear where the error originates: it could mean you used a newer kernel with older btrfs-progs with your BTRFS FS.
The Symptom
The mount fails on the new system, and dmesg throws a cryptic error:
corrupt leaf: block=830717952 slot=86 extent bytenr=2327262031872 len=36864 invalid data ref objectid value 18446744073709551604
The Cause: Metadata “Evolution”
This isn’t actually hardware corruption. It’s a validation disagreement.
The massive number in the error (18446744073709551604) is a technical “wrap-around” value for internal Btrfs references. This usually happens when a filesystem is created or managed using older btrfs-progs (like v5.16) on a system that doesn’t quite align with modern kernel defaults.
Newer kernels (6.0+) have a much stricter Tree-Checker. While the older system ignores these non-standard metadata values, the newer kernel sees them, flags them as “invalid,” and refuses to mount the drive to prevent potential data loss.
Mitgation: Read-only Rescue Mount
You can mount skipping all modern checks – but only-do read-only or there be dragons!
[!WARNING]
By using
rescue=all, you are telling the kernel: “Ignore the Tree-Checker.” The kernel will stop complaining about the invalid data ref objectid and allow the mount. However, because you are mounting in rw (read-write) mode, the kernel will attempt to update the metadata. If it writes new data based on a “leaf” it already thinks is weird, it could technically propagate that “weirdness” to other parts of the tree.
$ mount -t btrfs -o rescue=all,ro /dev/btrfs-volume /mnt/mountpoint
But note the messages DMESG:
[ +0.017638] BTRFS info (device dm-2 state CS): disabling log replay at mount time
[ +0.000004] BTRFS info (device dm-2 state CS): enabling disk space caching
[ +0.000001] BTRFS info (device dm-2 state CS): ignoring bad roots
[ +0.000001] BTRFS info (device dm-2 state CS): ignoring data csums
[ +0.000001] BTRFS info (device dm-2 state CS): ignoring meta csums
[ +0.000000] BTRFS info (device dm-2 state CS): ignoring unknown super block flags
The Fix: Rewrite Metadata
Since the old system can still read the drive, your data is safe. To bridge the gap:
On the old system which is able to mount, mount the volume and run a metadata balance:
$ mkdir /mnt/bad-drive
$ mount /dev/vda /mnt/bad-drive
$ btrfs balance status -mconvert=dup --bg /mnt/bad-drive
Done, had to relocated 13 out of 2816 chunks
$ watch -n1 btrfs balance status /mnt/bad-drive
You can use a VM to boot an older system via KVM and attach your block storage:
$ aria2c -j8 https://releases.ubuntu.com/focal/ubuntu-20.04.6-live-server-amd64.iso
$ sudo qemu-system-x86_64 \
-enable-kvm \
-m 2G \
-cpu host \
-cdrom ubuntu-20.04.6-live-server-amd64.iso \
-drive file=/tmp/cloud-init-RLrcc/seed.iso,format=raw,media=cdrom \
-drive file=/dev/mapper/btrfs-drive,format=raw,if=virtio \
-boot d
This forces the kernel to rewrite the “leaves” in a way that often cleans up these legacy references.
Best: The “Clean Slate” (Best Practice)
But still following could happen on a simple write to the fixed (?) FS …
[ +0.000000] BTRFS error (device dm-2): block=6363830763520 write time tree block corruption detected
[ +0.010717] BTRFS: error (device dm-2) in btrfs_commit_transaction:2540: errno=-5 IO failure (Error while writing out transaction)
[ +0.000003] BTRFS info (device dm-2 state E): forced readonly
[ +0.000005] BTRFS warning (device dm-2 state E): Skipping commit of aborted transaction.
[ +0.000001] BTRFS error (device dm-2 state EA): Transaction aborted (error -5)
[ +0.000001] BTRFS: error (device dm-2 state EA) in cleanup_transaction:2027: errno=-5 IO failure
I hope it’s not just a rumour that BTRFS has now reached maturity with Linux 6.x or next 7.x .
BTRFS is forward-compatible and I usually trust kernel-developers. IMHO if you are moving to a modern kernel permanently, the safest bet is to back up your data, reformat the drive using the modern system’s tools, and move the data back. This ensures the filesystem structures are native to the newer, stricter standards.
BTRFS is evolving fast. Sometimes, the “corruption” is just a newer kernel being a more observant librarian than the old one.
Always keep a backup and decent snapshots when working with BTRFS.
Recent Posts
BTRFS-ism: Metadata “Evolution” – Your Backup Drive is Dead!
TL;DR if you are looking for a quick solution check BTRFS-ism: The “Corrupt Leaf” Mystery!
The Scenario
If you didn’t know what BTRFS-ism is: I found myself stuck today when I plugged in my external USB backup drive:
$ sudo mount /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8 /media/backup
mount: /media/backup: can't read superblock on /dev/mapper/luks-5c3931a2-4333-415f-8e58-b3b4d728f4d8.
LUKS was opened successfully, but BTRFS refused to mount. Yes, it is BTRFS – again :(
Mise en place!
mise en place! Et de l’aqua de table, s’il vous plaît !
Still using asdf to manage your project- or home directory-binaries? Or you have nothing else in place than Dockerfiles? – It’s time to put that aside!
mise (pronounced “meez”) started as version manager like asdf and shortly emerged into a “polyglot” tool manager. It aims to be the only tool you need for your dev environment – therefore a “frontend” which you use to initialise a project with. It manages not just binaries, but also language runtimes (Node.js, Python, Ruby), environment variables – or as it’s three pillars state:
13 Shells of Christmas
Murex, Elvish, oil, nushell, es and ngs. I’ve heard the best parts of them do get adopted by Ion?
Infojunk December 2025
Random news and links to cool projects
-
Microsoft uses React Native in Windows 11 Start Menu – wow
https://pukhanov.ru/posts/windows-react-native?ref=dailydev -
IPtables again
https://www.booleanworld.com/depth-guide-iptables-linux-firewall/ -
Bichon is an E-Mail Archiver written ♥ Rust
https://github.com/rustmailer/bichon -
Prometheus-based Kubernetes Resource Recommendations
https://github.com/robusta-dev/krr -
The Lost Art of Progra,ming Offline
https://app.daily.dev/posts/the-lost-art-of-programming-offline-ksz0ysvkn -
GhosTTY Is Now Non-Profit which prevents it from a rug pull (aka. exit scam)
https://mitchellh.com/writing/ghostty-non-profit?ref=dailydev -
GhostTTY has now a xterm.js compatible web version
https://github.com/coder/ghostty-web -
AWS finally let’s you find idle NAT gateways
https://www.lastweekinaws.com/blog/aws-finally-lets-you-find-your-idle-nat-gateways -
Signs you work at a toxic company
https://www.blog4ems.com/p/signs-you-work-at-a-toxic-company
Help! We've ran into a DockerHub rate limit!
About
Yes, it is still happining. In 2025! Here you will find:
- Podman Dockerhub Mirror Configuration
- K8s Quickfix: Rewriting Existing K8s Resources
- Permanent Mirror Configuration for
containerd - K8s Admission Webhook to do the same
Podman Dockerhub Mirror Configuration
~/.config/containers/registries.conf.d/dockerhub-mirror.conf:
[[registry]]
prefix = "docker.io"
insecure = false
blocked = false
location = "public.ecr.aws/docker"
[[registry.mirror]]
location = "mirror.gcr.io"
[[registry.mirror]]
location = "gitlab.com/acme-org/dependency_proxy/containers"
[[registry.mirror]]
location = "registry-1.docker.io"
[[registry.mirror]]
location = …Fix for Amazon SSM Login on rootfs (no disk space left)
Sometimes AWS’s SSM StartInteractiveCommand doesn’t work once an EC2 instance’s root fs has run out of disk space.
aws ssm start-session \
--target i-0ab4e6dce100a0f58 \
--document-name AWS-StartInteractiveCommand
/etc/systemd/system/var-lib-amazon-ssm.mount
# SSM agent won't be able to login when disk is full
# so we reserve some space in-memory
[Unit]
Description=Mount /var/lib/amazon/ssm as tmpfs for SSM Agent
Documentation=man:systemd.mount(5)
Before=amazon-ssm-agent.service
[Mount]
What=tmpfs
Where=/var/lib/amazon/ssm
Type=tmpfs
# Mount options:
# defaults: Standard options
# noatime: Do not update inode access times for performance
# nosuid: Do not allow set-user-identifier or set-group-identifier bits to take effect
# nodev: Do not interpret character or block special devices
# noexec: Do not allow execution of binaries
# mode=1777: Set directory permissions to rwxrwxrwt (sticky bit, world-writable)
# size=64M: Limit the size of the …Amazon Linux 2025 has been officially canceled
(Some people didn’t take note of that, so let’s make it clear)
Already in 2024(!) AWS re:Invent had news on that topic that did not get much attention: There will be NO AL2025! *
AWS cited customer feedback requesting more stability and longer support cycles rather than frequent major version changes. Many organizations found the biannual major releases challenging for “enterprise deployment cycles”. – maybe 20th century organizations? 😄
Therefore:
- AL2 EOL has been extended to 2026
- and AL2023 EOL until 2029!
Instead of Amazon Linux 2025, AWS will focus on AL2023 Enhancement:
Craftsmanship And The Right Tools for Your Job
I wonder, why I my hardware was always superior than the one my companies provided me with. Shouldn’t they be interested in getting best quality? Would I order an electrician, and then forbid him to use his tools and give him my IKEA toolbox? – I wouldn’t!
A Software Craftsman’s tools are not mere instruments; they are his accumulated skill, capital, and tradition made tangible. I believe you have to care for your tools. Also, ★nix craftsmen often tend to solve problems with the capabilities of their systems.
DwarFS vs. SquashFS
I couldn’t see if DwarFS was comparing against SquashFS with LZMA. So I did my own tests.
Results
| Time | Size | Options | Details | |
|---|---|---|---|---|
| mksquashfs | 52.991 s ± 1.483 s | 213932 | -comp zstd -Xcompression-level 22 | zstd:level=22 (all) |
| mksquashfs | 50.122 s ± 0.871 s | 199332 | -comp zstd -Xcompression-level 22 -b 1048576 | zstd:level=22 (all) |
| mkdwarfs | 61.023 s ± 1.207 s | 182864 | –compress-level 7 | zstd:level=22 (block/meta) zstd:level=12 (schema) nilsimsa inode order |
| mksquashfs | 41.936 s ± 0.630 s | 213944 | -comp zstd -Xcompression-level 19 -b 1048576 | zstd:level=19 (all) |
| mkdwarfs | 42.884 s ± 1.007 s | 188816 | –compress-level 5 … |
Multi, Mono, Meta, Manifest – Composite Repository?
There was this discussion about whether to use Mono- or Multi-Repositories? I won’t pick it up again.
Some cool people suggested: why not use the best of both worlds and use meta-repos?!
I Was Interested in “Meta-Repo” Tooling
I asked myself how these tools would solve problems and gave some of them a try:
- There are multi-repository management tools that work by tagging or grouping repos, like gr or mu-repo.
- Tools that just do subdirectory iteration over your repos, like gitbatch.
- Many git-extras repositories with subdirectory iteration scripts that do the same, like git multi (the one I am using ;)).
- Tools which combine multi-repo manifests with different VCS systems, like myrepos (old and mature, often found within your Linux system package management).
- Tools that try to standardize the directory layout for managing your repositories, like ghq (you should definitely use such a layout!).
- Tools which basically reassemble what
git submoduledoes, like mu-repo, git-metarepo, or meta …