This guide will cover how to use the zfs send and zfs receive (or zfs recv) commands to migrate a filesystem from one system to another.

Basic Migration

At its simplest, the zfs send and zfs recv commands take a zfs snapshot and transfer it to the destination host. This allows these commands to be run on live, mounted datasets (at the cost of high bandwidth utilization). In this guide, I will use dest.cc as the system where the filesystem is being sent to and host.cc as the system where the filesystem originates from. Likewise, host-pool and dest-pool will be used for the zfs pool name.

1. On the host system, create a new manual snapshot of the filesystem you will send. I like to include the date in any manual snapshot
   1. zfs snapshot host-pool@manual_YYYY-MM-DD_migrate1
   2. You can check out current snapshots with zfs list -t snapshot host-pool
2. Create a new ssh key pair with a strong password and unique name for authentication off the dest system
   1. sudo ssh-keygen -t ed25519 -f /root/.ssh/dest_migrate_ed25519
   2. When prompted for a password, set a strong password, and don't leave it blank!
   3. If either system does not support ed25519 keys, you can use -t rsa with the -b 4096 arg as a substitute
3. On the destination system, add the public key of the key-pair to the authorized_keys file for root
   1. e.g., cat /root/.ssh/dest_migrate_ed25519.pub
   2. e.g., echo "ssh-ed25519 AAAAC3Nza[...]wgij root@host" >> /root/.ssh/authorized_keys
4. On the dest system, create any parent zfs filesystems for your desired final destination.
   1. e.g., If I want to the received filesystem to live under a new migration child file system, I would first have to zfs create dest-pool/migration
5. On the host system, launch a tmux window
   1. tmux
   2. You can detach a tmux session with {Ctrl + b} then d
   3. Reattach a session with tmux a [-t session_name]
   4. https://devhints.io/tmux
6. Now you're ready to send! Create your zfs send command and pipe it into an ssh command which then runs the zfs recv command
   1. e.g., zfs send host-pool@manual_YYYY-DD-MM_migrate1 | ssh dest.cc.gatech.edu -i /root/.ssh/dest_migrate_ed25519 zfs recv -s dest-pool/migration/my-moved-fs
   2. The -s option in the zfs recv part of the command allows you to resume an interrupted or failed zfs send (more below)
7. You will be asked for the password for the ssh key (you did set that, right?)
8. Once started, the zfs send command will not show any output or progress; however, on the dest system, you can watch the progress by doing zfs list dest-pool/migration and watching its size increase.
9. The zfs send command does not produce any output once complete, but you should get your cursor back.
10. Verify that the filesystem size you see on the destination matches your expectations.
11. Done!

Resuming an Interrupted or Failed Send

If for any reason a zfs send does not complete, you can pick back up on the transfer assuming that you have passed the -s option in the zfs recv command.

1. On the dest system, get the resume token by running zfs get receive_resume_token dest-pool/migration/my-moved-fs
2. Back on the sender, attach or open a tmux session
3. Run the same command as earlier; however, replace the entire zfs send portion of the command with a zfs send -t token
   1. e.g., zfs send -t 1-ed8dbba50-[...]af2563 | ssh dest.cc.gatech.edu -i /root/.ssh/dest_migrate_ed25519 zfs recv -s dest-pool/migration/my-moved-fs
4. If the send fails again, you will need to get the new receive_resume_token from the dest system

Sending Incremental Snapshots

The zfs send command is great because it sends a snapshot of a filesystem, allowing you to send live and in-use filesystems. What happens if data on the host has changed and you want to update your dest with the new content? You can use the -i and -I options to send updated snapshots to the dest.

The zfs filesystem on the dest filesystem cannot be modified after the initial send if you wish to send more snapshots. The zfs recv command cannot handle a diverging filesystem. If you wish to send more snapshots, you must first rollback the dest filesystem to the last snapshot that the sending host system knows.

Automatic sanoid snapshots on the dest file system should not affect the ability to send new snapshots so long as the data on the dest system has not changed. To help prevent unintended modification, you can unmount the filesystem with zfs set mountpoint=none dest-pool/migration/my-moved-fs

1. On the host system, create a new snapshot with zfs snapshot host-pool@manual_YYYY-MM-DD_migrate2
2. Follow the same steps under Basic Migration for creating keys and opening tmux if needed
3. Send the new snapshot up through the last snapshot the dest system knows about. This will also send any intermediary snapshots created on the host system (e.g., sanoid.timer snaps).
4. zfs send -I host-pool@manual_YYYY-DD-MM_migrate1 host-pool@manual_YYYY-DD-MM_migrate2 | ssh dest.cc.gatech.edu -i /root/.ssh/dest_migrate_ed25519 zfs recv -s dest-pool/migration/my-moved-fs
   1. Note that the first snapshot immediately following the -I is the earlier/older snapshot that the dest system last knows about. 
5. Depending on how much data has changed since the last send, this zfs send should complete faster since it's only sending the changes to the zfs filesystem since "migrate1" was created