Thirty Seconds of Nothing

For about a month my ThinkPad would stop responding for thirty seconds at a stretch. Not a crash (the pointer still moved, and I could sometimes coax a window switch out of it) but nothing repainted. Then it would come back on its own, as if nothing had happened. I had a theory about why, I had built mitigations around that theory, and the theory was wrong in an instructive way.

The machine is a P14s running Fedora 44, kernel 7.1.x, btrfs on LUKS, with 58 GiB of RAM. Everything below happened between the 6th and the 18th of August.

The Obvious Suspect

The freezes had started around the same time I changed my Borg backup job, a coincidence that feels like evidence. Backups are I/O-heavy, freezes look I/O-shaped, and I had touched exactly one thing. Case closed.

So I did what you do, and told the backup to get out of the way. The service got Nice=19, CPUWeight=20, IOSchedulingClass=idle and IOSchedulingPriority=7: the full "yield to interactive work" set, applied twice as it turned out, because I had put them in a drop-in as well as the unit itself.

The freezes continued exactly as before. That should have been the moment I questioned the premise. Instead I spent a while assuming I had applied the tuning wrong. A much more comfortable belief to hold.

Recording Instead of Guessing

What finally settled it was giving up on reasoning and building a recorder. The constraint is obvious once stated: any tool that writes to disk while diagnosing a disk stall becomes part of the problem. So the sampler wrote to tmpfs, did nothing per tick but read /proc/pressure/*, and fired the expensive per-process scan only when I/O pressure crossed 65 or something was already stuck. It ran as a systemd user service at Nice=-5 with OOMScoreAdjust=-500, so it would survive both the freeze and a suspend/resume cycle.

Two freezes was all it took. A representative sample:

io=s:58/f:57  mem=f:0.00  cpu=s:0.00  load=10.6,5.6,2.6  swap=8.0G
D=13[ btrfs-transaction  systemd-journal
      kworker/u64:*+btrfs-endio-write (×8)
      gnome-shell:wait_current_trans  gsd-color:wait_current_trans ]

Read that middle section again: CPU pressure zero, memory pressure zero. The machine was not busy. A load average of 10.6 alongside an idle CPU means every one of those tasks sits in uninterruptible sleep, waiting on something that isn't computation.

And the wchan column says exactly what they're waiting on: wait_current_trans. Thirteen tasks (including GNOME Shell itself) parked inside the btrfs transaction-commit path, behind eight writeback workers.

This is the textbook signature of a transaction-commit stall. btrfs is copy-on-write with a single filesystem-wide transaction; one enormous write-and-delete lands, its commit takes seconds to minutes, and every subsequent task that touches the filesystem queues up behind it. The desktop doesn't freeze because it's slow. It freezes because it's waiting in line.

Where the Writes Were Coming From

With the mechanism identified, the trigger fell out of journalctl in about a minute. Each stall was immediately preceded by a resume from suspend, and each resume kicked off GNOME Software applying updates in the background: a staged offline RPM transaction of roughly 9.5 GB, plus a batch of Flatpak and libostree pulls with their attendant old-ref deletions. One massive write-and-delete burst, delivered into the filesystem least equipped to absorb it gracefully, at the exact moment I had opened the lid and wanted to use the machine.

Borg, meanwhile, was thoroughly exonerated. Its on-resume run had failed in 18 seconds with exit 81 (network unreachable, because I was on a phone hotspot and the backup server wasn't routable). It was dead a full two minutes before the freeze began, and it appears in none of the captures. I had spent weeks tuning a process that wasn't running.

Why the Tuning Never Had a Chance

This is the part I find useful, because those mitigations weren't lazy. They were aimed at the wrong layer, and no amount of tuning them harder would have worked.

• The stall happens at the btrfs transaction-commit layer, which sits above the block-I/O scheduler. No I/O-priority scheme can reorder around a commit, because by the time you're in wait_current_trans the scheduling decision is already behind you.
• IOSchedulingClass=idle was doubly useless. ionice is only honoured under BFQ, and this machine's NVMe queue uses none, so the setting was inert regardless.
• Nice and CPUWeight address CPU contention, and the captures show CPU pressure at zero. Wrong resource.

Match the fix to the layer where the problem lives. I had three tools in hand and all three operated somewhere the problem wasn't.

The Fix, and a Small Lie from dconf

The primary fix is one line: stop GNOME Software downloading and staging updates in the background, and apply them by hand instead (on AC power, when I'm not mid-task).

gsettings set org.gnome.software download-updates false

Worth a warning, because it briefly convinced me the setting hadn't taken. Immediately after that command, gsettings get returned false while dconf read came back empty and the key was absent from a full dump. That's a transient service-cache artifact that hadn't been flushed to the on-disk user database; an explicit dconf write reconciled the two. If you've ever set a GNOME preference, verified it a different way, and concluded you were losing your mind, this is that.

Better, But Not Cured

Freezes dropped from nearly-every-resume to occasional. That is the right outcome, and worth saying precisely. I had removed the dominant trigger. The underlying mechanism was untouched. Any sufficiently large write burst can still serialise the desktop behind a single commit, and the same signature kept turning up now and then from other sources.

So on the 10th I added the seatbelt I had declined a few days earlier:

# /etc/sysctl.d/99-dirty-writeback.conf
vm.dirty_bytes = 268435456           # 256 MiB
vm.dirty_background_bytes = 67108864 # 64 MiB

The arithmetic is the argument. With the default vm.dirty_ratio=20 on 58 GiB of RAM, up to about 11.6 GiB of dirty pages can pile up before the kernel forces writeback, and btrfs then commits that entire backlog as one long transaction. Capping the backlog makes writeback start early and commit in small pieces.

Three notes on that drop-in. Setting the byte form makes the ratio knobs read 0, which is expected (they are mutually exclusive). The cap is moderate on purpose, since Pop!_OS once shipped an over-aggressive value that measurably degraded btrfs throughput, and 256 MiB is well clear of that. And it is filesystem-agnostic, helping interactive responsiveness under heavy writeback on ext4 and xfs too.

That last property mattered when I codified it. The task is gated on whether the host has a desktop, not on the filesystem being btrfs. I considered a btrfs gate and rejected it for three reasons: the tunable is a global VM setting and semantically about desktop interactivity, the cap benefits other filesystems anyway, and such a gate would switch itself off at the worst possible moment if I ever migrated. Per-host control belongs in an explicit toggle.

Would a Newer Kernel Fix It?

The obvious question, and I checked rather than assumed. The real upstream fix for this class of stall ("btrfs: do not block starts waiting on previous transaction commit", commit 77d20c685b6b) merged in 6.5, and this machine runs 7.1.x. It is already present, which reframes the whole thing as a workload-and-configuration problem.

I then went through the full 7.2 btrfs changelog looking for anything that would help. Nothing does. The direct-I/O de-serialisation work is a different path, since these stalls are buffered writeback. The writeback bio-size cap is the closest item, but it smooths the block layer, one level below where these tasks block. Large folios change the page cache and leave the commit model alone. And there is no snapshot-deletion work at all, which matters because snapshot churn is my leading hypothesis for the residual.

One warning if you're on btrfs and tempted by 7.2: avoid rc1 through rc6. The COW fixup worker was dropped early in that cycle and caused silent data loss before being restored in rc7. Stable 7.2 is fine.

Why I Am Still on btrfs

Would moving to ext4 or xfs fix this? Yes, unambiguously. Both journal metadata with fine-grained commits and have no whole-filesystem copy-on-write transaction for everything to serialise behind. A large write there is just slower I/O.

I'm not doing it, and I should be clear about what buys the reprieve. btrbk takes hourly and daily snapshots, which gives me instant local rollback and is also what my backup job ships, so that would stop working. Home directories run compress=zstd:1. And Incus container storage is a btrfs subvolume with copy-on-write clones, which on ext4 or xfs falls back to a much slower directory driver with no fast snapshots.

There is also no in-place conversion. It is back up, reformat, restore. So the decision is to exhaust the cheap reversible mitigations, identify the residual trigger, apply something targeted (thinning snapshot retention is far cheaper than reformatting), and treat migration as a last resort arrived at with data rather than irritation.

What I Would Tell Myself in July

Four things, in order of how much time each would have saved me.

Capture beats inference. I had a plausible, coherent, wrong story, and I had built real infrastructure on top of it. Two captured events destroyed it. The narrative felt like knowledge and wasn't.

The wchan column is the money signal. Knowing that tasks are stuck is nearly useless. Knowing where in the kernel they are stuck points straight at the layer to fix and rules out every other layer at the same time. One column of ps output did more than weeks of theorising.

Instrument without perturbing. Sample to tmpfs, gate the expensive work behind a pressure threshold, and run at elevated priority so your recorder isn't the first casualty of the thing it is recording.

Verify tunables against the actual hardware. That IOSchedulingClass=idle looked like it was doing something for weeks. It was a no-op the entire time, and one look at /sys/block/nvme0n1/queue/scheduler would have said so.

The freezes are mitigated and I'm still watching. If they come back in earnest the harness gets re-armed, and this time I will believe what it records instead of what I expect.