The Fault Follows the Port
In June I declared a 10 TB drive dead. It had thrown link errors, timed out on reads, and ZFS had marked it FAULTED after six read and seven write failures. I wrote it off in my notes, ordered a replacement, and ran the pool degraded for two months. In August I discovered the drive was in perfect health and had been the whole time. The fault was a five-dollar SATA port on a cheap PCIe card, and it would have faulted anything I plugged into it.
That is the story worth telling, because it is the clearest instance of a pattern I have hit repeatedly since building this server: ZFS kept reporting hardware problems accurately, and I kept misattributing them. The filesystem was never the unreliable narrator.
What Is Actually in the Box
The machine runs Ubuntu LTS with ZFS 2.2.2 and 31 GB of non-ECC DDR4, with boot and root on an NVMe under LVM. The OS is disposable; everything that matters lives on ZFS. Three pools:
data— four striped mirrors (three of 2× 10 TB, one of 2× 4 TB), about 30.9 TB. Datasets for home directories, media and container state, with LZ4 compression; the home dataset is ZFS-native-encrypted.apps— three striped mirrors of SSD and NVMe, about 1.59 TB. All the container and application data, including a Postgres database.backups— two mirrors in an external enclosure, about 10.9 TB. A local replica and nothing else.
Mirrors everywhere, never RAIDZ. For a pool serving containers and a database, mirrors give better IOPS and resilver dramatically faster, but the reason that actually decides it is growth: I can grow the pool two drives at a time. On a home budget, incremental growth beats storage efficiency.
The Fault Follows the Port
Back to the drive. The June symptom was ata10.00: SError: { DevExch } plus
READ FPDMA QUEUED timeouts, appearing on the first boot of a new kernel. My note at
the time read: "the new kernel's SATA driver likely tipped over a marginal drive/cable." That is a
reasonable-sounding sentence, and every part of it's wrong.
What eventually cracked it was an accident. During a night of swapping hardware I ended up shuffling drives between ports, which handed me a controlled experiment I hadn't planned:
- The surviving drive from that mirror flapped continuously on
ata10(link resets, timeout counters climbing) and then came up perfectly stable the instant it moved to a motherboard port. - A drive that had been rock-solid for years dropped straight to FAULTED the moment it landed on
ata10.
The fault stayed with the port regardless of which drive was in it. That is about as clean a result as home diagnostics ever produce.
There was also a signal I had all along and hadn't read properly. SMART on the flapping drive
was clean, with healthy media and (crucially) a CRC error count of zero. CRC errors are what you
get when the signal is bad, so they point at the cable. Zero CRC errors combined with
DevExch and command timeouts means the link is failing to negotiate at all, which
points at the port or the PHY. I had bought new SATA cables and a fresh power lead chasing this,
and neither changed anything, because the number that would have told me not to bother was already
sitting in smartctl output.
Root cause: a cheap ASMedia PCIe SATA card with at least two dead ports (one that faults anything attached to it, and one that will not link at all).
The rebuild was an LSI 9207-8i in IT mode with every spinning drive moved onto it, plus a
proper power supply to replace the one whose shortage of SATA leads had produced a nest of
splitters. Every drive came up stable at 6 Gb/s with zero flapping. And the drive I wrote off in
June, once reinstalled on the new controller, was recognised by ZFS from its old mirror label and
auto-resilvered back in (no zpool replace required). It reads SMART PASSED with zero
reallocated, pending, or CRC errors.
The only dead drive in the whole saga was the replacement I'd bought. It spins up but presents no SATA link on any known-good port, which I proved by swapping its lead with a working drive's and watching the dark port follow the drive.
Growing a Pool You Cannot Shrink
I added three vdevs this year, and every one followed the same ritual, because zpool
add is one of the few unforgiving commands in ZFS. Vdev removal exists mostly to undo an
accidental add; you don't get to reshape a pool afterwards. The topology has to be right the first
time.
So: wipe the candidate drives properly, reference them by stable ID, never
/dev/sdX, match ashift to the existing vdevs, and always dry-run
first.
sudo wipefs -a /dev/disk/by-id/<drive>
sudo sgdisk --zap-all /dev/disk/by-id/<drive>
# -n first: confirm it reports a mirror, not two single-disk vdevs
sudo zpool add -n -o ashift=12 apps mirror /dev/disk/by-id/<a> /dev/disk/by-id/<b>
sudo zpool add -o ashift=12 apps mirror /dev/disk/by-id/<a> /dev/disk/by-id/<b>The dry run guards one specific catastrophe. Omit the mirror keyword and ZFS
cheerfully stripes two single drives into your redundant pool, permanently, with no redundancy and
no way back short of destroying and rebuilding. The -n output tells you which of
those two things you're about to do.
A note on wipefs, since it caught me out: it isn't always enough. One drive
carried a corrupt EFI label that wipefs left behind and zpool add
rejected, and sgdisk --zap-all cleared it. I now run both as a matter of course.
Two things about mixed-size vdevs surprised me, both benign. Striping a 4 TB mirror alongside three 10 TB mirrors works fine, since capacity is additive and ZFS biases new writes toward the roomier vdev. But existing data does not rebalance. A freshly added vdev stays cold until new writes arrive, or until you force the issue with a send/receive round trip. If you add capacity expecting the pool to even itself out, it won't.
Off-Site Without Handing Over the Keys
The home dataset is ZFS-native-encrypted and replicates off-site to rsync.net, which means trusting someone else's disks with data I'd rather they couldn't read. Raw send is what makes that comfortable:
zfs send -wR <pool>/secure@snap1 | ssh <host>.rsync.net \
"zfs recv -u -o readonly=on -o mountpoint=none data1/backups/secure"The -w flag sends the encrypted blocks as-is, so the remote stores ciphertext and
never holds the key. You can confirm that from their side, where zfs get
encryption,keystatus should report on and unavailable. Never load
the key there, not even once to test something; the entire guarantee is that the passphrase stays
home.
Two details I would carry anywhere. First, the replication is a pull, meaning the remote host logs in here and pulls snapshots rather than this host pushing to it. That way a compromise of my server yields no credentials that can reach out and destroy its own backups.
Second, the replication user's permissions. Those started as
send,snapshot,hold,destroy,mount,rollback, which I had granted without much thought.
I tested what that actually allowed, and found the remote puller could destroy arbitrary snapshots
(and whole datasets) on the source. It is now send,hold,release, granted per
replicated dataset.
And there's a trap in there: release is a separate permission from
hold. Grant hold without it and every replicated snapshot accumulates a
hold that can never be cleaned up, quietly pinning storage forever.
When a Missing Pool Takes the Whole Machine
The worst outage I have had wasn't data loss at all. In April the USB enclosure holding my local backup pool failed, with all vdevs going UNAVAIL simultaneously (which is enclosure-level failure, not drive failure). Minutes later the monthly scrub cron fired on schedule.
A scrub on a pool with no vdevs took the ZFS namespace lock and held it for 34 days.
By the time I looked properly, load average sat around 157 with the CPU 64–76% idle, since
every one of those tasks was in uninterruptible sleep. A zpool list from a month
earlier was still running. Kernel threads for USB disconnect had been wedged for roughly a hundred
days. Both udev and journald were jammed. Even dpkg --configure -a appeared to hang,
because its initramfs hook calls zpool and joined the same queue.
Nothing in userspace could fix it. Processes in that state cannot be killed, every ZFS command needs the held lock, and even a clean reboot hangs waiting on the stuck daemons. The way out was to stop the pools auto-importing on next boot, then force a reset through sysrq:
sudo mv /etc/zfs/zpool.cache /etc/zfs/zpool.cache.bak
echo s | sudo tee /proc/sysrq-trigger # sync
echo u | sudo tee /proc/sysrq-trigger # remount read-only
echo b | sudo tee /proc/sysrq-trigger # reboot nowLoad went from 157 to under 1 and the entire stack cleared. The lost pool was only a local replica, so nothing irreplaceable went with it.
The rebuilt pool has one property the old one didn't:
failmode=continue
The default is wait, which blocks callers indefinitely when a pool becomes
unavailable, and that is what turned a dead enclosure into a month-long host outage. Setting
continue returns errors to callers instead. For a pool that lives in a removable
enclosure and holds nothing primary that is obviously the right trade, and I would argue it is the
right default for any replica target.
What the Failures Had in Common
Looking back over the year, the ZFS layer was right every single time and my interpretation was the weak link.
ZFS reports a symptom; it can't tell you a cause. "Drive FAULTED" is accurate and it says nothing about whether the fault is the drive, the cable, the port, the controller or the power. I read a mechanical conclusion into a status field that was only ever describing I/O outcomes.
Corroborate across layers before condemning hardware. The clean CRC count, the passing self-test, and a scrub repairing nothing all pointed away from the drive. Any one of them in isolation is weak; together they were decisive, and I had all three in June.
A non-zero counter is not a rising counter. A related lesson from a different incident: my monitoring had been mailing daily about a drive with 32 offline-uncorrectable sectors. Those sectors haven't moved since January, across roughly 5,700 power-on hours. The alert fired on the value being non-zero rather than on it increasing, and an alert that cries wolf daily for eight months trains you to ignore the one channel that matters.
USB is fine for a replica and nothing else. I still run a USB-attached backup pool, on purpose, because its worst failure costs me a reseed and nothing more. But it took the whole host down once, and the durable answer for anything you actually depend on is a SAS JBOD behind an IT-mode HBA.
The pools are all online as I write this, the scrubs repair nothing, and the drive I declared dead in June is resilvered back into its mirror and working perfectly. Which is a good outcome arrived at badly, and the two-month detour was my own doing.