This guide covers testing a salt branch before manually advancing a PR into the next development phase.

System Selection

Before running manual tests, you should choose several systems that represent your current production environment. At the simplest level, this means selecting systems representing different operating systems (e.g., Ubuntu, RHEL), but this can be broadened to include systems like GPU systems, non-x86_64 systems, file servers, etc. The changes you are testing should help dictate what system types are appropriate. For example, if implementing a change to ZFS backups, you should ensure RHEL systems and file servers are included in your testing suite.

Testing Production

After selecting your systems, you should test that they work as intended with the production environment. This allows any deviations found to be attributed to the system itself rather than the incoming changes. Template states should not be applied to machines unless the existing configuration for that state can be overwritten. If your incoming changes fix a known issue that causes production to fail, you should note that in a comment, but not necessarily under the Quality Deviations section of your PR. The following is a good start to the steps in testing production; however, any features or changes to production since this guide was updated should also be accounted for.

Update and Sync

1. Run sudo salt-call saltutil.sync_all saltenv=coc-prd to sync the current production behavior
2. Fully patch systems with apt update && apt upgrade or yum update
3. If possible, reboot if patches were applied

Apply States

If a state fails, rerun the state.apply command. If it fails again, the failure should be noted in the Quality Deviations section of your PR. If appropriate, an Issue should be submitted.

1. Run sudo salt-call state.apply saltenv=coc-prd pillarenv=coc-prd
2. Run sudo salt-call state.apply util.motd_template saltenv=coc-prd pillarenv=coc-prd
3. Run sudo salt-call state.apply util.sshd_template saltenv=coc-prd pillarenv=coc-prd
4. Run sudo salt-call state.apply root_authorized_keys saltenv=coc-prd pillarenv=coc-prd
5. Run sudo salt-call state.apply fail2ban saltenv=coc-prd pillarenv=coc-prd
6. Run sudo salt-call state.apply sssd saltenv=coc-prd pillarenv=coc-prd
7. Run sudo salt-call state.apply coc-base.sudo saltenv=coc-prd pillarenv=coc-prd
8. If appropriate, run the sudo salt-call state.apply coc-base.pam saltenv=coc-prd pillarenv=coc-prd state
9. Run any production states that have been introduced since this list was updated or that are relevant to the incoming changes

Check System Status

Check system status for expected behavior. If any of the below checks deviate from the expected behavior, please either 1) fix the issue and try again, 2) use a different host for testing, or 3) note the deviation either in a comment or under the Quality Deviations section of your PR.

1. Ensure Cortex is running: sudo systemctl status traps_pmd
2. Ensure Qualys is running: sudo systemctl status qualys-cloud-agent
3. Ensure fail2ban is running: sudo systemctl status fail2ban
4. Check that you can log in as yourself (sssd check)
5. Check that you can log in through a bastion such as nevah or carora (root_authorized_keys check)
6. When attempting to log in, ensure that the Georgia Institute of Technology - Terms of Use banner was displayed (sshd_template check)
7. When logged in, ensure that a MOTD is displayed (motd_template check)
8. Check sudo privileges: sudo id
9. Ensure apt update or yum update works
10. Check GPUs if applicable: nvidia-smi
11. Check ZFS is applicable: zpool status and zfs list
12. Check any services that have been introduced since this list was updated or that are relevant to the incoming changes

Testing Incoming Changes

Following your testing of production, you are now ready to test your incoming code changes. Template states should not be applied to machines unless the existing configuration for that state can be overwritten. The following is a good start to testing incoming changes; however, any features or changes to production since this guide was updated should also be accounted for.

The salting and pillarenv in this example shows "coc-dev"; however, you should change this to whichever git branch hosts your incoming changes!

Update and Sync

1. Run sudo salt-call saltutil.sync_all saltenv=coc-dev to sync the current production behavior
2. Fully patch systems with apt update && apt upgrade or yum update
3. If possible, reboot if patches were applied

Apply States

If a state fails, rerun the state.apply command. If it fails again, the failure should be noted in the Quality Deviations section of your PR. If appropriate, an Issue should be submitted. If the failure matches a failure observed in your testing of production, that should also be noted in the PR.

1. Run sudo salt-call state.apply saltenv=coc-dev pillarenv=coc-dev
2. Run sudo salt-call state.apply util.motd_template saltenv=coc-dev pillarenv=coc-dev
3. Run sudo salt-call state.apply util.sshd_template saltenv=coc-dev pillarenv=coc-dev
4. Run sudo salt-call state.apply root_authorized_keys saltenv=coc-dev pillarenv=coc-dev
5. Run sudo salt-call state.apply fail2ban saltenv=coc-dev pillarenv=coc-dev
6. Run sudo salt-call state.apply sssd saltenv=coc-dev pillarenv=coc-dev
7. Run sudo salt-call state.apply coc-base.sudo saltenv=coc-dev pillarenv=coc-dev
8. If appropriate, run the sudo salt-call state.apply coc-base.pam saltenv=coc-dev pillarenv=coc-dev state
9. Run any production states that have been introduced since this list was updated or that are relevant to the incoming changes

Check System Status

Check system status for new behavior. If any of the below checks deviate from the expected behavior, please either 1) fix the issue and try again, 2) use a different host for testing, or 3) note the deviation either in a comment or under the Quality Deviations section of your PR.

1. Ensure Cortex is running: sudo systemctl status traps_pmd
2. Ensure Qualys is running: sudo systemctl status qualys-cloud-agent
3. Ensure fail2ban is running: sudo systemctl status fail2ban
4. Check that you can log in as yourself (sssd check)
5. Check that you can log in through a bastion such as nevah or carora (root_authorized_keys check)
6. When attempting to log in, ensure that the Georgia Institute of Technology - Terms of Use banner was displayed (sshd_template check)
7. When logged in, ensure that a MOTD is displayed (motd_template check)
8. Check sudo privileges: sudo id
9. Ensure apt update or yum update works
10. Check GPUs if applicable: nvidia-smi
11. Check ZFS is applicable: zpool status and zfs list
12. Check all new behavior that has been introduced in the code changes
13. Check any services that have been introduced since this list was updated or that are relevant to the incoming changes

Last updated: Oct 15, 2024
Last updated by: Garrett Briaud