Development Tips / Intro
Thanks for interest in working on Jet's source code! Understanding some of the following topics may make for a better development and contributing experience. Also consider reading Contributing if you haven't already!
Whether you are familiar with Rust or not, we're glad to have you.
Just as early devops tools exposed new people to Python and Ruby, we also hope Jet helps develop an interest in Rust among new audiences. Rust is largely a systems language, and sometimes development is slower, but Rust also reduces the time spent finding errors, such that written code often sometimes works the very first time like magic. We love it for this.
A few concepts occur fairly commonly in Jet code and warrant review - you may be able to pick these up just by skimming the code - but may also want to read about them elsewhere online.
- 1.Arc is a smart pointer to a value on the heap, and Arcs are used extensively throughout the program. Various values are not able to be passed around the stack or as references, particuarly with the concurrency goals of the program in SSH mode.
- 2.We use Mutex and RwLock to manage concurrent access to objects. Mutex allows for exclusive read or write access to one thread at a time. RwLock allows access to any number of readers but only one writer. We use RwLock unless a Mutex is specifically required. We use Mutex around SSH Connections and occassionally to keep output together, but that is mostly it.
- 3.As with most Rust programs, we use Result<OkType,ErrorType> to indicate a value may return either a given result or an error. Similarly Option is used to represent a return value that may be None.
- 4.We rather heavily use match for case-like statements, especially when working with Result and Option results. There are a lot of shortcuts in Rust like unwrap_or and such, but often match is the clearest, and it works for everything, so we prefer using match in almost all cases.
- 5.We use a lot of Rust collections like HashMap and HashSet, as well as iterators like map.
- 6.We use a fair amount of Enums. We often use Enums instead of booleans for better clarity when reading code that calls a function that would take a boolean as a parameter, this way we don't have to remember what true means for the third argument to a function call. The list of valid modules in Jet that you put under "tasks:" is implemented as an Enum.
- 7.The question mark operator "?" is used to send the error portion of a result up to the current function when an error occurs, allowing for more concise code when the error has the same return type as the outer function itself. For this reason, we try to keep the return types the same in most cases. The Rust compiler will warn you when a result is ignored, so this is not something you will likely forget to leave out, but it is always worth being in the mindset of thinking "what am I doing with the errors from this function?"
If you have any Rust questions at all, or just want to talk about Rust, stop by the #rust channel in our chat. There are no bad questions!
Regardless of development background, a few particular design details about the program are important to note:
- 1.Inside of modules, both Ok results and errors often use an object called TaskResponse ('src/tasks/response.rs'), usually shared with an Arc and generated by the TaskHandle. There's a fair amount of code to convert errors into TaskResponses. Noticing this pattern will help module development make sense, because you will not be able to return results/errors of other types inside the main module file.
- 2.Inside of modules, 'src/tasks/handle.rs' (TaskHandle) provides user access to most common functions and ensures consistency between modules. Think of this as the "power tool" interface to making modules easy. Using Task Handle also makes sure modules stay on the right track and have common conventions. Various features of task handle are namespaced into other structs hanging off of task handle, such as template functions for processing parameters, or 'response' for shortcuts around creating return objects.
- 3.Outside of modules, such as when parsing CLI arguments, most errors are simple strings.
- 4.The best way to understand the internal control flow of the entire program is to start with main.rs and notice that based on the "cli/cli_parser.rs" code, various functions in main.rs are selected that behave differently. For CLI modes involving playbooks (most of them), common traversal code in "playbooks/traversal.rs" is used to walk over the playbook tree, and these are set up in cli/playbooks.rs. Execution of tasks, including parallel execution of SSH tasks, lives in "tasks/task_fsm.rs" - and this is what makes the "dispatch" functions of modules work. These are all of the guts of the system before you get to modules, which are the easy part that can remain blissfully oblivious of everything behind the scenes! See the module development guide for module info as this will be the most frequently adjusted part of the program!
- 5.Inventory is a critical concept for SSH management only, but there is still a "localhost" object and "local connections" that are used for local management options. This local management is still used in SSH operations for some internal tasks, like identifying local checksums of files for the copy module. Some familiarity with connections may be useful. When you see references to "remote", even in local context, it refers to the connection object. Local always refers to the machine running 'jetp'.
- 6.The Rust code is almost never allowed to call panic! except in cases of a module being coded incorrectly according to the contract, or code that is essentially impossible to execute without a major coding error in the program. Panic would stop all tasks vs just failing the current thread. Module execution in SSH is distributed among many threads, we return an Err(TaskResponse) instead when we encounter an error situation. This allows us to report the failure and continue on other hosts. Sometimes you will see panic in some places of the code that are impossible to reach conditions that need some code there to satisfy the compiler.
With that out of the way, see the other sections in the docs for details about other aspects of the program. If your question is still unanswered, hop by Discord and we'd be glad to explain and talk about anything.
Editor choice is up to you. Michael uses a completely stock VSCode install as he finds some of the overlays from popular Rust plugins distracting. The "rainbow" colored parens in VSCode can be useful when dealing with some nested quasi-functional-programming expressions.
- 1.Remember that all style and preferences are subjective. That being said, to get contributions accepted you need to turn off auto formatting in your editor and do not run the code through tools like rustfmt. Reformatting an entire file is to be avoided. Generally try to match the style of the rest of the program, which tries to be a bit more concise than rustfmt.
- 2.Make sure all code is free of rust compiler warnings on compilation. Compile by running "make" in the root of the source checkout.
- 3.We generally do not care about warnings from tools like clippy. Fixing some of them may be nice but we disagree with others, so leave these changes up to project leadership and do not include them in pull requests. We prefer explicit return statements.
- 4.Try to make the code as clean, clear, concise, and self-explanatory as possible. Interior comments on difficult to understand code are welcome, especially when the Rust compiler wishes to make things look a little strange.
- 5.Think about future developers other than yourself and make things easy for them. Choose high quality variable names and function names. Matching Ok(x) is more than fine as is sometimes using "x" on an iterator, but it is especially important to give "let" variables descriptive variable names. There is no need to comment things that would be obvious to those that know rust, or to explain the purposes of function names that are already self explanatory, but a one or two line comment per function never hurts.
- 6.Consider the error paths, return types, and security implications of your code very carefully. This will be a major part of code review. The Rust compiler helps here a lot though!
- 7.Try very hard to avoid introducing new crate dependencies, especially those that are not widely adopted. Introduction of new depedencies is always going to be a needs-chat-discussion point. The good news is, dependencies in module development in particular are seldom useful, as in SSH mode we do not push code or programs out to remote machines. Where we can interface with a stable OS CLI tool, that is always preferred. Since all code in SSH mode runs on the control machine, we cannot use dependencies to effect change directly on remote systems, even though that would technically work well in local configuration modes, it is not allowed to have module features that only work locally.
- 8.See the Contributing notes in full and understand all of the items. Per our Contributing page, we strongly suggest joining Discord chat if you wish to submit a feature or something more than a simple bugfix.
- 9.There are no wrong questions! Ask questions if you want to understand how anything works or get stuck in understanding something! We enjoy conversation and want to hear from you, and also want to make your experience with jet fun and successful.