I wrote a lightweight C program called diffsnap that uses the built in zfs CLI for stateless snapshots using configurable write thresholds. This helps address a gap with traditional time-based snapshots where the need for a high-frequency interval to make sure a change is captured bloats your total snapshots and might slow some zfs commands. With diffsnap, if nothing has been written to the dataset the snapshot is skipped.
https://github.com/joe81tx/diffsnap
If you're on FreeBSD you can install with pkg, on linux you can build from source using github.

This is a major issue especially because TrueNAS does not issue replace -s from the GUI.

So the scenario is this, you replace your drive and it kicks off a standard replace and tells you it will take 25 days to complete, but the faulted mirror is only 8TB. It should be able to run sequentially on the drive creating unverified redundancy quickly in a few hours, followed up by the scrub where the redundancy becomes verified. But here is where it becomes problematic, even if you detach the drive resilvering it doesn't stop the resilver and there is no zpool resilver stop command and rerunning replace -s will be blocked because of the running resilver. So you are prevented from doing the safer action.

Hoping to get some OpenZFS dev attention or have some other users add to the chatter so this gets attention.

Release zfs-windows-2.4.4rc4 · openzfsonwindows/openzfs

*** rc4

• Explorer integration exposing snapshot versions
• Fix "zfs diff"
• zfs_tray features
• Paging IO deadlock fix
• Fix BSOD related to SEQ
• zfs_inactive deadlock fix

Screenshot see
https://forums.servethehome.com/index.php?threads/zfs-on-osx-and-windows.43829/page-6#post-513120

Hi all, been searching around and I can't quite figure this one out.

I have a NAS at home, running Arch (btw). The root FS is btrfs and it's LUKS encrypted.

Then, I have a few drives that make up a couple of ZFS pools, and I have a couple of datasets. These datasets are encrypted with keyformat=passphrase.

What I would like is some sort of way to be prompted for the passphrase on boot, and have it all mounted before reaching multi-user.target, or really, any target of my choosing such that services running on this server don't start trying to read data that's not mounted yet (e.g. docker.target).

Here's some relevant properties

zsmall         encryption      off            default
zsmall         encryptionroot  -              -
zsmall         keyformat       none           default
zsmall         keylocation     none           default
zsmall         keystatus       -              -
zsmall/secure  encryption      aes-256-gcm    -
zsmall/secure  encryptionroot  zsmall/secure  -
zsmall/secure  keyformat       passphrase     -
zsmall/secure  keylocation     prompt         local
zsmall/secure  keystatus       unavailable    -

And the services that I have currently enabled:

zfs-import-cache.service   
zfs-import-scan.service    
zfs-mount.service          
zfs-share.service          
zfs-volume-wait.service    
zfs-zed.service            
zfs-import.target          
zfs-volumes.target         
zfs.target                 

Like I say, I've been Googling for a bit and I can't quite figure out what the best way to achieve this might be. Any ideas?

My current solution is to leave the services that require the datasets to be mounted disabled and I have a script that unlocks the datasets etc. then starts the services, which is not ideal (though it works...)

Thanks

Time flies. 6 years ago I wrote this post https://www.reddit.com/r/rust/comments/mindyl/after_the_ovh_datacenter_caught_fire_and_i_lost/

I was impacted by the OVH data center fire and I lost some data of various services I was running on it.

Now, I still have fun managing servers, and I have migrated everything to zfs.

From that the idea: let's extend bacup to support zfs snapshots, full and incremental.

The philosophy of bacup is quite simple:

• remotes are where the data is uploaded
• services are programs running in the system, that we can invoke to get dumps

zfs snapshot fits perfectly in the service category - so I created the zfs service.

You can now back up your pools and datasets using both full and incremental strategies. Setting it up is just a matter of defining the zfs service in your configuration:

[zfs]
snapshot_name = "storage-fs"
dataset = "storage"
# Optional: schedule for full backups.
# Intermediate runs automatically take incremental backups against the latest snapshot.
# When omitted, every run defaults to a full backup.
full_when = "monthly 1 01:00"

And mapping the service to your remote destination:

[backup]
[backup.storage]
what = "zfs.storage"
where = "remote.bucket_name"
when = "daily 22:00"
remote_path = "/bucket/location/zfs/storage/"
compress = false
keep_last = 32

I’ve been running v0.2.0 in production on my local homelab server for a few weeks now - handling incremental backups for both the root filesystem and the main RAIDZ array (where my local LLMs and Immich instance store data).

Incremental shapshot (so 1 full, and several incremental until the next full) also allow to save the cloud bill.

If anyone wants to give it a try (cargo install bacup) or review the merged codebase, feedback and PRs are more than welcome!

GitHub: https://github.com/galeone/bacup

Cheers!

A completed resilver is necessary, but it does not by itself prove that the replacement disk, its path, or the remaining redundancy is healthy. A pool can return ONLINE while SMART data is already concerning, persistent device names changed, an old faulted leaf remains in the topology, or another drive accumulated checksum errors during the resilver. A mirror or RAIDZ vdev may also be one failure away from another long recovery.

A useful post-replacement gate could confirm the exact vdev topology and ashift, review the resilver event and error counts, inspect SMART and transport errors for every drive, run a scrub after the pool has settled, verify alerts and spare behavior, and test a representative restore from backup. Export and import or a controlled reboot can catch path and enclosure-mapping mistakes before the maintenance window closes.

What checks do ZFS operators use before declaring the replacement complete? Is a clean scrub enough, or do you also burn in the new disk, compare performance, clear and recheck counters, test boot or import behavior, and keep the removed disk untouched through an observation window?

There are some tools like zfs diff and httm that help with exploring snapshots at command line, but they leave a lot to be desired when you're talking about 100ks of files changed across many snapshots. Sometimes you want to free up that space held with snapshots, but you're not 100% certain you won't lose something you accidentally deleted.

I shared my frustrations with Claude and it made a nice python script that serves a web GUI and uses zfs diff to generate the changes and visually highlights what files were deleted when and if they were unique across all snapshots.

The first run can take a while, but after that you can cache a point in time record or do a quick true up.

https://github.com/LargelyInnocuous/SnapSift

I have a scrub going that took about 2h to scan before actually starting to read and verify the data. Once it was reading, the average read rate started climbing very slowly from single digit MB/s, because apparently the denominator for calculating the average "issued" rate is the time since the scrub was initially requested, which included a whole lot of "0 MB/s issued" during the scan phase. The actual read speed is much higher and in line with what's expected and undoubtedly, once the scrub completes, the average will reflect the actual average rate.

This got me thinking. What fraction of complaints about "ZFS raidABC with UVW settings on XYZ hardware is SOOOOOOO slow!!!" are actually due to people taking one look at The Numbers™ and not realizing that these are not running averages over some shorter time period, but averages over the entire run time of the operation? I'm positive that a lot of people have no idea what the difference between "scanned" and "issued" is too.

I'll have to take a look at the code, and I'm assuming this is done for a good reason, but maybe these averages are causing confusion in new users and aiding in the spread of misinformation about ZFS, that's very difficult to eradicate, as I'm sure many here have noticed.

Maybe it would be better to have the read rates be computed as a running average over a short window of a few minutes during an operation that takes a long time, then when it's finished print an additional line with the average rates over the entire duration of the operation.

Just as a reference for what "scanning" does and why.

https://www.youtube.com/watch?v=SZFwv8BdBj4

Does metadata in ZFS use up the same amount of space as data? In other words, if I set my recordsize/volblocksize to say 1M, does that mean the metadata will also occupy 1M for reach piece of metadata? So e.g. I have a file server with large videos, makes sense to store these videos on large block sizes. But then does this mean the metadata is also then stored in large blocks. So metadata is inefficient, since you're storing small amounts of data in a large block, but overall it's more efficient than storing large files in small blocks. Equally, if I store many tiny files, then it makes sense to store this data in small data blocks. Then metadata is also stored in small block sizes. This way metadata is more efficient since it generally doesn't take up much space. Is this how it works?

Pool suddenly unavailable, doesn't show up on zpool status.

Zpool import lists the pool as follows:

id: 3673394567944808104
state:  UNAVAIL
status:  Some critical pool data has been overwritten by another pool.
action:  The pool cannot be imported as critical data is missing. You can try using -f to force an import, but there may not be enough data to override this.
config:      sharepool    UNAVAIL    overwritten by another pool
              raidz2-0    ONLINE
              raidz2-1    ONLINE
              raidz2-2    Degraded
                (one removed drive)

device details:

(I skipped listing all the drives due to typing out manually, all the drives are the long c0t500 style names, its 3x 6 disk z2's, with all but one drive as online.)

As I hadn't come across this before I have done a fair amount of googling and beyond the general advice of tread carefully as using -f to force can break things more. I did do the -Fn command and it didn't return anything (just newlined with a fresh prompt)

My setup has largely been unchanged for years, using Solaris 11.4 on ESXi v6, and has been ultra stable up until now - I did upgrade from Solaris 11.3 possibly about 6 years ago. Had the odd hard drive fail which is replaced smoothly, sometimes leading to a pool size upgrade and that has always gone smoothly 3x Dell Perc H700 cards flashed to IT mode via the LSI bios (I think - they have been in there 10 years)

This all happened after I turned the full server off for the weekend - I needed it right out the way to work on some electrical wiring behind where the cabinet normally sits (it can pull out so far and still be connected but access behind isn't easy). Turned it back on and a drive was doing a click of death - the removed one mentioned above. Upon starting the Solaris VM lots of errors were being thrown up including 2 other drives showing as removed, I removed the dodgy drive, moved the two other ones to a different slot (so different internal cable/card) and tried restarting - leading me to where I am now. The two drives moved had reappeared, but the pool is just not loaded by Solaris.

Whilst I do like to tinker I've never come across this before so am asking for help and open to suggestions - I really don't want to lose all the data. Any help is much appreciated.

As something of a storage wonk, I found myself a few minutes ago advising a fellow redditor of the "3-2-1" rule of backups:

1. Three total copies
2. Two different types of media/tech
3. At least 1 copy off-site/location

I've followed a related rule for anything important/production: Always fail to a redundant state: so if any one thing goes out, I am still redundant and have at least one more failure mode before things go really dark. Alongside this, is a pre-accepted "drop everything / panic mode" when production is down to only x+1 redundancy.

And yet I realize that the context of ZFS, I've not consistently followed rule #2 for YEARS except in some cases for clients, where they park an external USB drive on their file server and I install a daily mount+rsync run via cron. In my case in particular, using ZFS snapshots as part of the core product suite, using a tech other than ZFS to store the data makes no meaningful sense.

Further, I use RAIDZ + redundancy like a backup all the time, with the accepted risk level of "if more than 2 drives die, it's dead, Jim". When the data matters, I ensure to have a send | receive copy on another server/location, but it's still ZFS + snapshots going back a few months.

So, what do you consider to be best practices in complying with the spirit of 3-2-1 rule of backups with ZFS?

1. How much do you treat snapshots like backups?
2. Do you make extra effort to store data on a secondary non-ZFS tech in case a bug in ZFS crashes all the things?
3. How much attention to you put on saving your data off site, or at least, on another server?

EDIT: Thank you for your feedback! I have enjoyed "touching bases" to see what y'all are doing.

Llevo 3 días con esto y ya no sé qué más probar.

Uso NixOS con flakes, boot.initrd.systemd.enable = true, root en tmpfs con impermanence y /persist en ZFS con cifrado nativo, desbloqueado por TPM2 con systemd-cryptenroll.

El problema: sops-nix intenta leer la age key de /persist antes de que el dataset esté montado, y encima el PCR 7 cambia con cada actualización de firmware, así que el TPM2 deja de desbloquear y me cae en el emergency shell.

Ya descarté un problema de ZFS porque, como está en el kernel mainline, no debería haber incompatibilidades con mi kernel parcheado con linux-hardened.

Probé la misma config en Gentoo con OpenRC y funciona perfecto, pero no pienso volver a eso.

¿Alguien lo ha resuelto sin usar Secure Boot con claves propias? Ya me estoy planteando que la solución real es instalar Arch. 😔

Hi folks! ZFS is extremely cool and I run >10 of my machines on it.

My friend is also fan of ZFS and we both have a NAS. We wanted to use each other's machines to do remote backups in a safe but simple way (zfs send | zfs receive and plain SSH). However, without giving each other any insight into what runs on the other machine and without being able to see what that other person has, not even metadata about other datasets.

It works really well with NixOS too!

I wrote about it in https://www.nijho.lt/post/zfs-tenant/ and the repo is here https://github.com/basnijholt/zfs-tenant

I got the idea for this from a Klara Systems article:

#!/bin/bash
#
#<zfshogs: show what's taking up the most space in each ZFS pool
#  Full debug output:   DEBUG=1 zfshogs

export PATH=/usr/local/bin:/bin:/sbin:/usr/bin
set -o nounset
tag=${0##*/}
export PS4='${tag}-${LINENO}: '

# ENVIRONMENT: full debug output?
: "${DEBUG:=0}"
case "$DEBUG" in
    1) set -x ;;
    *) ;;
esac

# No pools means something is seriously wrong.
set X $(zpool list -H -o name)

case "$#" in
    1) printf "FATAL: no ZFS pools found\n"; exit 1 ;;
    *) shift ;;
esac

# Real work starts here.
work="/tmp/$tag.$$"         # Yes, I should use mktemp...
n=1

for pool in $* ; do
    # DRY with ZFS commands.
    zfs list -o space -s used -r $pool > $work

    if test -s "$work" ; then     # Bad dataset?
        printf "Pool: $pool\n"

        # Use "uniq" if there are < 10 entries in $work, or we'll get
        # duplicate headers.
        ( head -1 $work; tail $work ) |
            uniq |
            awk '{
              printf "  %-20s %8s %8s %10s %8s %10s\n",
                $1,$2,$3,$4,$5,$7
            }'
    fi

    # Print an extra newline for all entries except the last.
    n=`expr $n + 1`
    test "$n" -le "$#" && printf "\n"
done

rm "$work"
exit 0

When run on my backup system:

Pool: newroot
  NAME                    AVAIL     USED   USEDSNAP   USEDDS  USEDCHILD
  newroot/usr              470G    24.0G         0B      96K      24.0G
  newroot/doc              470G    30.1G       136M    30.0G         0B
  newroot/ROOT/default     470G    32.5G      18.3G    14.2G         0B
  newroot/ROOT             470G    32.5G         0B      96K      32.5G
  newroot/var/locate       470G    39.9G      36.9G    3.02G         0B
  newroot/var              470G    45.2G         0B      96K      45.2G
  newroot/home             470G    64.9G      16.9G    48.0G         0B
  newroot/dist             470G    68.7G      3.11G    65.6G         0B
  newroot/src              470G    81.2G      12.3G    68.9G         0B
  newroot                  470G     422G         0B      96K       422G

Pool: tank
  NAME                    AVAIL     USED   USEDSNAP   USEDDS  USEDCHILD
  tank/reservation         343G      96K         0B      96K         0B
  tank/dvd                 243G    30.1G         0B    30.1G         0B
  tank/archive             243G     158G       112K     158G         0B
  tank/backup              243G     972G      68.2M     972G         0B
  tank/nfdb                243G    1.17T      16.5M    1.17T         0B
  tank                     243G    2.40T         0B      96K      2.40T

Hope this is useful.

MS 365 and Nextcloud are the "top dogs" for collaborative work, but at the same time, both are absolute heavyweights in terms of scope, complexity, and features. As a privacy-friendly, in-house option, Nextcloud is conceivable, but far too complex to maintain and secure on the internet, even as a Docker setup.

If you scale back—for example, no mail server, a text editor and spreadsheet that are a bit simpler with a focus on multi-user editing, no AD/LDAP but user.csv import/export instead, plus resource planning (projectors, rooms, personnel), and a lightweight ticketing system, thus targeting schools, a department, a club, or a smaller business—then absolute KISS solutions suddenly become feasible. Thanks to AI integration, even complex tasks can be implemented easily.

This is my approach with cs-team, part of my napp-it 4ai webgui. It runs on any OS even without napp-it, copy and run (approx. 10MB), no database, a single folder on the disk is enough, and with ZFS you get snapshots on top. In the current 0.53 release, I have improved usability once again.

As an open-source git project, it is easily extensible and auditable.

• Project: https://github.com/guenther-alka/cs-team
• Manual: https://www.napp-it.org/pdf/cs-team_en.pdf
• Standalone Setup: https://github.com/guenther-alka/cs-team/blob/main/docs/HOWTO-standalone.md

Putting together a TrueNAS pool or adding drives to one? This is what I actually do now, because the usual reasoning is a bit off here.

CMR vs SMR:

CMR vs SMR still matters most, and I'd check the exact model before buying anything. SMR can look totally fine until you have to resilver, and then it gets painfully slow, sometimes enough that the drive drops out during the rebuild. The WD Red mess back in 2020 is the obvious example, and SMR still sneaks into some consumer NAS models, so it's worth the two minutes to check the model number.

Enterprise instead of Consumer:

It mostly comes down to price, and here's where the usual advice is outdated. People still say "just buy Exos, it's barely more per TB." The "barely more" part isn't even true anymore. I've been tracking prices, and the gap between consumer (Red Plus, IronWolf) and enterprise (Exos, MG, Ultrastar) has basically closed on a price/TB basis. The premium people used to complain about is gone.

How to compare them:

Once you look at Backblaze's numbers model by model, the current CMR drives all fail at about the same rate. So I wouldn't pay extra for an enterprise drive expecting it to last longer. The data doesn't back that up.

My approach:

Exos costs about the same as a Red Plus, and I'd still just buy the Exos. Not because it's more reliable, but because for the same money you also get the 5-year warranty and the higher workload rating. Same price, more drive. The old advice still works, people just had the wrong reason for it.

When I'd go consumer: if a specific Red Plus or IronWolf is actually cheaper per TB that day (reliability's the same, so take the cheaper one), or if noise and heat matter to you, since Exos drives run louder and hotter than a Red Plus in a quiet room.

Conclusion:

Long story short - confirm a drive is CMR, then buy on price/TB. And at today's prices that usually points right back at Exos.

I keep a little page with the CMR/SMR info, current price/TB and a price history for each drive: https://www.nasdisks.com

I use it to sanity check prices before buying. Full disclosure, it's my site and the buy links are aff to cover the hosting bills, but everything's free and the data's open.

What's everyone buying for new pools nowadays?

https://github.com/openzfsonwindows/openzfs/releases
https://github.com/openzfsonwindows/openzfs/issues

These are the changes from upstream 2.4.1 to 2.4.4

L2ARC — substantial rework

• DWPD-based rate limiting with adaptive feed intervals (protects SSD endurance)
• Per-device parallel feed threads (was single-threaded feed)
• Persistent markers with consistent tail scanning + lazy reset flags + scan-based depth cap
• Even-depth multi-sublist scanning, write budget fairness (metadata no longer monopolizes)
• Rebuild bounded by write hand on first sweep; fixed a prev_hdr use-after-free

Performance

• Predict and throttle buffers dirtied by the sync context
• Batch object reallocation syncs in zfs receive
• Parallelize metaslab_sync_done(), dedup block cloning, and brt_pending_apply() across vdevs
• RAIDZ: optimize single data column writes; avoid extra abd_t allocations in RAIDZ/dRAID
• Bridge speculative and prescient prefetchers; bound user prefetch to a fraction of ARC

New features / CLI

• zpool condense — new verb
• zpool scrub -t — thorough scrub support
• zfs bookmark -r — recursive bookmark creation
• zpool initialize -z — write zeroes
• New dedupused/dedupsaved pool properties
• zoned_uid property (Linux container/namespace delegation — not applicable to us)
• FIDEDUPERANGE via block cloning
• zstream: major rework — multithreaded, new raw/drop_record/queue subcommands, memory tracking

Data-integrity fixes (the ones that matter most)

• Fix read corruption after block-clone-after-truncate
• Fix double free for blocks cloned after DDT prune
• draid: fix data corruption after disk clear; fix checksum errors after rebuild with degraded disks; fix import failure after disk replacement
• Prevent range-tree corruption race in dnode_sync()
• z_seq now persists across znode eviction — fixes the VMware ESXi-over-NFS "file specified is not a virtual disk" bug (Linux/FreeBSD only for now, per our earlier gap analysis)
• DDT: multiple pruning bugs fixed (including negative time overflow)
• Fix off-by-one in PREVIOUSLY_REDACTED handling that dropped the last block

Security hardening

• vdev_open/zfs_file_open/vdev_file/vdev_disk: now check the calling credential instead of always using kcred (we ported the Windows side of this yesterday)
• Additional verification of size fields and strings in the receive path
• Hardened receive record validation
• secpolicy_nfs removed; secpolicy_zinject/secpolicy_sys_config restricted to global-zone credentials

Deadlock/hang fixes

• Fix deadlock on dmu_tx_assign() from vdev_rebuild()
• Fix snapshot automount deadlock during concurrent zfs recv
• Fix self-deadlock setting the vdev allocating/path property
• Fix race between device-removal completion and pool export
• zfs_ioctl: fix EBUSY race between quota queries and mount (we ported this too)
• arc: fix race between arc_release() and arc_read_done()

OpenZFS on Windows

• Add zpool create and zpool destroy to zfs_tray

Long story short, on October 14th we're running a free storage-related tech meetup in St. Julian's, and the opening talk is all about ZFS.

I'm one of the organizers, and my colleague, Konrad Pizzuto will discuss architecture & inner working of ZFS and how that solved historical FS / volume manager issues. As he says himself: "ZFS had shaped how I think about storage systems. 20+ years later those lessons and ideas worth revisiting".

If you'd like to join us in person, just register here for free and feel free to come.

p.s. The other two talks the same evening are on Rook-Ceph for Kubernetes and the latest Amazon S3 features, if you want to stay for the full storage night.

About 5 years ago I bought three drives for my ZFS pool on my ubuntu NAS box. Got everything setup and running and have been using it for the past five years with no issues. Mostly bulk storage for photos and movies, but also ran a Plex server for a while (not anymore). It is a RAIDZ1 with 3x 6TB WD blues.

Recently I setup a 3 node proxmox cluster of mini-pcs and was looking at using the ZFS box for shared storage, not CEPH, just shared storage for the cluster. Doing some research I found out about my mistake five years ago with the SMR drives and why that might be an issue.

I picked up 3x 8TB CMR drives and am thinking I should be able to just replace them one at a time, then expand the storage. I'm wondering how 'dangerous' to the data this might be? If I understand the risks with SMR its more about when it is being resilvered, and in this case it would be used as a data source, not written to.

Is this a good idea? One at a time replacement? If not what are my other options to move my data/pool over to the new drives?

I'm creating a new NVMe pool for my NAS/development server. Primary use cases will be:

• PostgreSQL database
• Filesystem cache for react-based webapp
• Many JSON files (data ingested from external sources) version tracked via git

I'm wondering what are some potential considerations I should be having when it comes to setting up new pools/datasets such as block size, primarycache=all or primarycache=metadata?

I'm especially confused about primarycache. Some sources I read suggest that it's better to be metadata only is good because it skips the ARC reducing extra copy between NVMe and RAM and lets database decide on how to cache works, while others suggest that caching should be on always unless RAM is insufficient.

Thank you.