To create backup snapshots of ZFS host

On host:

Install sanoid using simon stream "sw-sanoid"

If you are using ubuntu 22 or Rhel 9 and cannot access simon, follow the instructions to manually install at https://tso.cc.gatech.edu/howtos/manually-installing-syncoidsanoid-linux-systems

The sanoid.conf file should include the below default template.

/etc/sanoid/sanoid.conf

#############################
# templates below this line #
#############################
  [template_production] frequently = 0
  hourly = 36
  daily = 30
  monthly = 3
  yearly = 0
  autosnap = yes
  autoprune = yes

Edit the sanoid.conf file to configure what is to be snapshotted. Add each zpool you want to have snapshots of avobe the template section in the sanoid.conf file. However this will get all filesystems on the zpool which is likely not what you want.

/etc/sanoid/sanoid.conf

[ZpoolName]
  use_template = production
  recursive = yes

To avoid backing up scratch, tmp, reserve or other unneeded snapshots cluttering the server you can list the file systems you want to keep individually. When you do that you must also include the base zpool but just change the recursive property to "no". Example below.

/etc/sanoid/sanoid.conf

[ZpoolName]
  use_template = production
  recursive = no

[ZpoolName/FSName]
  use_template = production
  recursive = yes

Since we still have the recursive option to "yes" for the file systems it will get any sub filesystems that exists. This is important especially for our home directories, allowing us to just include home and get all the FSs below.

To create remote copies of zfs snapshot backups

On host:

Make sure sanoid is set up as detailed above before working on syncoid.

Add cc-hereford's syncoid.pub key to the host at /root.ssh/authorized_keys. If the host is running simon, then the keys should have been added automatically.

On Backup Servers (cc-hereford & cc-braford)

Syncoid backups occur as a daily cron job that reference the files located at /etc/sanoid/syncoid-filesets-hostname.conf. Before creating the cron job, we'll want to run a single one time transfer. This is done both to make sure that transfers will happen successfully and not to bog down the system if the initial transfers takes a significant amount of time.

1. Create a copy of an existing filesets conf file in /etc/sanoid/ that is named for your system
   • e.g. cp /etc/sanoid/syncoid-filesets-volvo.conf /etc/sanoid/syncoid-filesets-HOSTNAME.conf
2. Modify the new file to include the appropriate filesets that are listed in the host's sanoid configuration. The [/FSName] is only required if you are stating those separately in your sanoid conf on the host.
   • nano /etc/sanoid/syncoid-filesets-HOSTNAME

   • SRCLIST=" \
     root@HOSTNAME.cc.gatech.edu:ZpoolName[/FSName] \
     root@HOSTNAME.cc.gatech.edu:ZpoolName[/FSName2] \
     "

3. Before setting any automatic backups, we want to do a one time manual pull with syncoid to the host in order to make sure that everything is running and not bog down the cron job. To do this, we'll create a softlink to the manual sync script in cc-hereford using the naming convention of the conf file in /etc/sanoid that was just created. The script uses the name of the softlink that is referring to it as the variable and needs to match accordingly.
   • e.g. ln -s /root/bin/syncoid-manual /root/syncoid-filesets-HOSTNAME
4. Save the file and run the script. It would be best to run this in a screen/tmux session. This process can take a long time depending on how much else is transferring on the backup server and the size of the filesystem you are backing up.
   • If you are having issues with asking for a password, make sure that the host has the syncoid key from cc-hereford
5. Once this initial backup is done, you can verify by checking the snapshots in cc-hereford for your system
   • (on braford) zfs list -d 1 brafordpool/syncoid-backups | grep HOSTNAME
   • (on hereford) zfs list -d 1 monsterpool/syncoid-backups | grep HOSTNAME
6. Once this is confirmed, you can add your system to the daily cron job as a softlink. Using the same logic as before, the name of the softlink is the variable that will be used to determine which sanoid conf file to use.
   • e.g. ln -s /etc/cron.daily/syncoid-filesets /etc/cron.daily/syncoid-filesets-HOSTNAME

If it is a VM server that is being backed up add to the /etc/cron.daily/syncoid-vms file instead.