Tracking Team Failure on Solo Minion
I have been using salt for the past month. Whenever I run a command say sudo salt '*' test.ping
then master pings all minions and the answer is a list of all minions that are up and running. The result looks something like this:
{
"minion_1": true
}
{
"minion_2": true
}
{
"minion_3": true
}
In the main file, conf
the return type is set to JSON. But if I execute the wrong command through the salt master say sudo salt '*' test1.ping
then the master will return something like this
{
"minion_1": "'test1.ping' is not available."
}
{
"minion_2": "'test1.ping' is not available."
}
{
"minion_3": "'test1.ping' is not available."
}
In both outputs displayed above, the command issued a success exit code on the main shell / terminal. How we keep track of which minions failed to complete commands. I am not interested in what error, I just need some way or another to track minions that could not execute the command.
The last solution is to write a parser that will read the full output and decide for itself. Hope there is a better solution.
source to share
Despair reasons
I won't rely on the Salt CLI exit code at this time (version 2014.7.5
) - there are still many issues open to resolve this.
Get valid JSON output
There is an option --static
that captures the JSON output:
If you are using
--out=json
, you will probably need--static
. Without the static option, you will get a JSON string for each minion.
Otherwise, the output above in Salt contains multiple objects (one per minion) which is not valid JSON (JSON requires a single object, array, or value per document) and an easy way to load all output with a standard JSON parser will fail. It is even mentioned in the documentation (c 5188d6c
):
Some JSON parsers can guess when an object ends and a new one starts, but many cannot.
In addition to this, some Salt options (for example show_jid
) also send strings to STDOUT, which mix it with the execution protocol and invalidate the JSON output format. The option --static
also solves this problem.
UPDATE: Analyzer to detect failure when executing Salt
This problem squeezed me so much that I quickly gave birth to this Python script @ 75e42af with an example of how @ b819961d uses it .
NOTE. This will not deal with the output of an arbitrary Salt command (including the test.ping
above), but issues related to the exit of execution status will be addressed. There is still a solution to the problem test.ping
above - it can be run from state, then the output can be parsed with a script. See how to call this execution unit from a state or *.sls
file in this answer .
Features (details in the code itself):
- Process output data from both
highstate
andorchestrate
. - Process the output of multiple minions and any number of commands.
- Summary of "? From N" report and overall result.
- A standalone file can be used as a script and a module.
The only limitation is that JSON output (Salt option --out json
) is required simply because it is easy to fix the issues discussed before loading them into the parser.
source to share