Current revision updated by dhoward32 on
Originally created by dhoward32 on

1) Context for this guide

This guide IS NOT meant to be a substitute of the official documentation of Salt that OIT and the Salt Project provides online. That documentation is linked below. This guide IS intended to be a brief overview of what Salt is, the core concepts that are moving, and a bit of how it relates to TSO usage specifically, as we, of course, do things slightly differently here than the rest of the world.

Feel free to link back to the landing page if this is not what you're looking for, or contact the salt team at TSO if there is something that is missing or unclear.

For additional documentation

2) What is Salt

Salt is a configuration management system. It's primary purpose is to be able maintain remote systems in a specifically defined states. For example, ensuring that configuration files maintain a specific set of content, packages are installed, etc. Salt is designed around the idea of remotely managing all of this through the use of admin interfaces, however, because all of our nodes can be pretty unique from each other, we tend to use a mixture of remote and local administration. At the time of writing, TSO is using Salt in all new deployments and managements of systems that have been deployed since the beginning of 2024. However, we are not supporting Salt for anything less than Rhel 8 and Ubuntu 20, and many existing systems have not been onboarded into Salt. As we go through upgrading older systems, we will be making sure to enroll them in Salt to get everyone under the same setup.

While we might know salt primarily by running basic commands like salt-call state.apply, there is a fair amount happening under the hood that this guide will attempt to cover at a base level.

3) Why is Salt

Primarily we are migrating to Salt in order to move on from the previous management system, Simon. Ideally, Salt is more supported than Simon as it is a full externally developed platform, similar to Ansible, Terraform, and Puppet. We are also using Salt over Simon as it is OIT supported and we can expected assistance from outside the college with maintaining of everything.

Using salt gives us the ability to centrally run bulk commands to any number of our systems simultaneously.

4) How do I access Salt?

  • Their regular account needs to be in the GRS group
    • gtaccountentitlement=/gt/departmental/coc/tso/coc-tso-coc-admins/enable
  • Must have an active PSA account that is able to access IAT with delegated admin access
  • PSA account must be set up with 2FA through passport
  • PSA account assurance level must be assurance leve 50 through IAT
  • The github.gatech.edu account for their gt account (not PSA account) needs to be set up using ssh keys
  • The github.gatech.edu account needs to be set up with 2FA
  • If you want access to kubernetes, you will also need your PSA account to have a primary email address

More information can be found at https://docs.az.gatech.edu/s/190ffd15-195a-4b2f-ae4a-9316559a3816/doc/verifying-access-0SrNCDdSUI

5) Salt Realms

We have somewhat coined the term salt realm, but this is what use to describe the tiers of states that are being run on the system when you execute a salt command.

  • Formulas
  • States
  • Pillars
  • Grains

These realms run in order from bottom --> top and higher level realms have access to the results of the lower level realms. For example, grains will always run first, and states will have access to the results of those grains. And similar, grains will not have access to pillar or state information.

Grains

Grains are a collection of dictionaries that render out details such as software install status, model info, etc. These run locally on the minion and generally system properties or custom attributes (is borg installed, what is the OS version, etc). Simplified, a grain is what the system knows about itself.

Pillars

Pillar is an interface designed to offer global values to minions. These are defined and managed by the salt master and usually used to define specific attributes that can be defined with grain logic, or "arbitrary" assignment (should this system have borg installed, what are the default access groups, etc). Simplified, a pillar is information that the salt master knows about the system, but the system doesn't necessarily know about itself.

States

A state is a representation of the state that a system should be in. This can be done in the form of installing packages, configuring a file, setting authorized keys, etc. A state can also reference other states. For example, running state.apply will actually run multiple states that we have configured to run. Simplified, a state is the actions that a system is performing with the knowledge it has from the grains and pillars.

Formulas

A formula is a collection of states that can be run as a group of tasks. Each formula is actually it's own git repository that can be linked into the main git repository. For example, this is how we reference the OIT defined states, Cortex and Qualys.

How are these written?

Github! Please refer to the code development lifecycle and guidelines page.

6) Minions

Minions are what salt refers for the salt instance on the server. We generally use minions and servers interchangeably as we name our minions the same as the system. The reason they are technically separate is that Salt encourages naming minions after groups and clusters for easier scripting (research-fung_cluster-1), but due to our setup with our servers, we have decided to keep the naming 1-1.

7) Admin Interfaces

Admin interfaces are what we refer to for any of the central interfaces that can send salt commands to minions. These allow us to remotely send commands, send bulk commands, or start scripting out commands. There are several interfaces that will be discussed in more details, but primarily we use the following

  • Local
    • Local is the primary way that we send salt commands. This is when you log in to a server and run a command that you want to run on that server.
  • Alcali
    • Alcali is the web interface that we use to send remote commands to servers and also accept registration keys of servers. Most often, we use this to accept the registration keys that are required for onboarding a server into salt. However, you can also use alcali to send salt commands through a web interface in a similar syntax to local commands.
  • Pepper
    • Pepper has very similar features to alcali, however, it is a CLI service that is hosted on our VM, spice-rack. Similar to alcali, this lets you send out remote commands to servers, but you get additional advantages of being hosted on a server and can create additional scripting and logging output.
  • Kubectl
    • Kubectl is an interface with talking directly with the Salt master that OIT hosts. This is primarily for getting diagnostics on the salt master itself that OIT hosts in a kubernetes container in Azure.

8) Salt Master

The Salt Master is a server that is running the salt-masterservice. It issues commands to one or more Salt Minions, which are servers that are running the salt-minionservice and that are registered with that particular Salt Master. We do not directly log in to the Salt master as it is hosted and managed by OIT, but this is the link between our admin interfaces and hosts.

Because it is hosted and managed by OIT, we don't have much input into the running or servicing of this unfortunately, and instead, submit servicenow tickets when things are broken.

9) How does Salt execute

So, you've gotten this far through all the terminalogy, but how does Salt actually run?

Say you log in to a server and run a salt-call state.apply command?

  • Interface - Local
  • Action - state.apply
  • Minion - ccg-wiki
  • Execution Realm - States (runs grains, then pillars first)

 

Back to top
Filing Categories
Identifier Categories
Specific categories