[juju] Add plugin option for Juju state reporting #3803

MichaelThamm · 2024-10-10T18:25:59Z

This change modifies the Juju plugin to optionally capture Juju state (controllers, models, applications, units) information. The capture assumes that juju is installed on the machine where sos is called, and that the juju user has superuser privilege to the current (or requested) controllers.

This option is disabled by default because, while the information can be very useful, the collection acts on the live Juju state.

Use the plugin with the feature:

sos report --only-plugin juju -k juju.juju-state=True

Run with a specific user (who has access to the controllers specified). This defaults to ubuntu:

-k juju.juju-user=super_admin

If necessary, you can filter by controllers or models with cluster options.

-k juju.controllers="controller_a controller_b"
-k juju.models="controller_a:model_x controller_b:model_y"

If nothing is supplied for either the controllers or models options, the report will include all state information for them respectively.

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

Is the commit message split over multiple lines and hard-wrapped at 72 characters?
Is the subject and message clear and concise?
Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
Does the commit contain a Signed-off-by: First Lastname [email protected]?
Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
Are all passwords or private data gathered by this PR obfuscated?
Add tests to ensure results are scrubbed / functioning

packit-as-a-service · 2024-10-10T18:28:40Z

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

sudo yum install -y dnf-plugins-core on RHEL 8
sudo dnf install -y dnf-plugins-core on Fedora
dnf copr enable packit/sosreport-sos-3803
And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

sos/report/plugins/juju.py

arif-ali

As well as turboturtle comments, can we also look at the flake/pylint issues as well is any updated PR

sos/report/plugins/juju.py

jcastill · 2024-10-11T19:06:03Z

A couple of very small comments from me.
The DCO is still failing, I think you are missing '-s' while running 'git commit'.
Last thing, please make sure you squash the two commits you have at the moment into one.

jcastill · 2024-10-13T17:44:09Z

@MichaelThamm remember to squash all the commits into one and sign it

MichaelThamm · 2024-10-14T17:29:54Z

@arif-ali or @jcastill I imagine that I need to obfuscate some of the following information which is included in the juju controllers command output:

        "lxd": {
            "user": "admin",
            "recent-server": IP:Port,
            "uuid": "ac03821c-361e-4daf-83c9-a712cd56b8d4",
            "api-endpoints": [IP:Port],
            "ca-cert": "-----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----",
        },

Is there a need to obfuscate these things or is it just passwords, tokens, keys as mentioned in obfuscate?

arif-ali · 2024-10-14T17:34:56Z

@MichaelThamm The IP addresses and endpoints, not so, as they would be typically handled by --clean or sos clean, if indeed these needed to be obfuscated.

In terms of the certs, you can get the example of the sunbeam.py plugin, where we just needs self.do_cmd_private_sub('juju controllers') in the postproc() function. This will automatically obfuscate any command that showed the cert.

sos/report/plugins/juju.py

MichaelThamm · 2024-10-15T14:18:55Z

@arif-ali

In terms of the certs, you can get the example of the sunbeam.py plugin, where we just needs self.do_cmd_private_sub('juju controllers') in the postproc() function. This will automatically obfuscate any command that showed the cert.

When I run: self.do_cmd_private_sub('juju controllers --format=json') I see this in the logs:

2024-10-15 10:13:54,360 DEBUG: [plugin:juju] substituting '-----SCRUBBED' for '----(?:-| )BEGIN.*?----(?:-| )END' in commands matching '*juju controllers --format=json*'

However, I do not see results in the commands dir of the sos report tar.xz file

This method does not have a filename parameter either. Not sure how to continue with this?

arif-ali · 2024-10-15T14:27:31Z

When I run: self.do_cmd_private_sub('juju controllers --format=json') I see this in the logs:
2024-10-15 10:13:54,360 DEBUG: [plugin:juju] substituting '-----SCRUBBED' for '----(?:-| )BEGIN.*?----(?:-| )END' in commands matching '*juju controllers --format=json*'
However, I do not see results in the commands dir of the sos report tar.xz file

This method does not have a filename parameter either. Not sure how to continue with this?

You can just specify juju controllers in the function, and it will do a match. In the sunbeam.py, I did the same with doing a self.collect_cmd_output("juju controllers --format json"), and then used the juju controllers in the self.do_cmd_private_sub()

Signed-off-by: Michael Thamm <[email protected]>

MichaelThamm · 2024-10-15T14:56:00Z

I think the PR is almost ready for review, but I noticed we have tests for the Juju plugin. Is there any need to update this since the test for scrubbing seems to be sufficient? I also tested the scrubbing for juju controllers manually and it passes.

TurboTurtle · 2024-10-15T22:28:51Z

sos/report/plugins/juju.py

+        )
+        if controllers_json["status"] == 0:
+            desired_controllers = set(
+                self.get_option("controllers").split(" ")


We generally have plugins use : as a delimiter in plugin options that can take a list (as the use of commas gets murky with argparse and how plugin options are....parsed.

The only problem with this is for the Juju model synatx it can be:

The current controller's model -> model_a

Or it can be a specific controller and model -> controller_x:model_a

For this reason I chose to split via empty space since models and controllers are not allowed to have spaces in their names according to Juju.

i.e. controller one -> not valid name

Would this be a reasonable exception to the Sos plugin standard?

If it's an expectation that users would pass something like -k juju.controllers="foo:bar", then yeah it's totally reasonable to use something else. We should just make note of that, ideally in both the plugin option description (for sos report -l output) and the docstring for the plugin (as the docstring gets used in sos help output, e.g. sos help report.plugins.juju - see the kernel plugin docstring for a good example).

TurboTurtle · 2024-10-15T22:31:36Z

sos/report/plugins/juju.py

+
+        # Specific models
+        if self.get_option("models"):
+            for model in self.get_option("models").split(" "):


Ditto, we prefer plugin options to use : as a delimiter. We may need to add this to the plugin guide, I'll check this shortly.

TurboTurtle · 2024-10-15T22:32:33Z

sos/report/plugins/juju.py

+        if self.get_option("models"):
+            for model in self.get_option("models").split(" "):
+                command = f"juju status -m {model} --format=json"
+                self.collect_cmd_output(command, runas=juju_user)


collect_cmd_output() is designed to return the output for further processing, if you don't need to do anything with the output, use add_cmd_output() instead.

TurboTurtle · 2024-10-15T22:43:19Z

sos/report/plugins/juju.py

+                            f"juju status -m {controller}:{short_name} "
+                            f"--format=json"
+                        )
+                        self.collect_cmd_output(command, runas=juju_user)


Same here, use add_cmd_output() if you don't need to do anything further with the output.

TurboTurtle · 2024-10-15T22:46:32Z

sos/report/plugins/juju.py

+        if self.get_option("controllers") and self.get_option("models"):
+            self._log_warn(
+                "Options: controllers, models are mutually exclusive. "
+                "Will not collect Juju information."
+            )
+            return


I'm assuming these are mutually exclusive within the Juju stack, but would we not anticipate some (admittedly niche) use case where an sos end user isn't sure if a particular host is using one or the other, and wants to pass e.g. -k juju.controllers=foo,juju.models=bar to get either without needing to know the specific host configuration? I'm mainly thinking of some sort of automation-driven sos collection here, but I could be way off with that idea.

TurboTurtle reviewed Oct 10, 2024

View reviewed changes

sos/report/plugins/juju.py Outdated Show resolved Hide resolved

arif-ali requested changes Oct 11, 2024

View reviewed changes

sos/report/plugins/juju.py Outdated Show resolved Hide resolved

jcastill suggested changes Oct 11, 2024

View reviewed changes

sos/report/plugins/juju.py Outdated Show resolved Hide resolved

sos/report/plugins/juju.py Show resolved Hide resolved

jcastill approved these changes Oct 13, 2024

View reviewed changes

MichaelThamm force-pushed the report-juju branch from 60d0c0e to 9d36d66 Compare October 14, 2024 15:30

MichaelThamm changed the title ~~Feat: Report on juju environment~~ [Juju] Add plugin option for Juju state reporting Oct 14, 2024

MichaelThamm changed the title ~~[Juju] Add plugin option for Juju state reporting~~ [juju] Add plugin option for Juju state reporting Oct 14, 2024

arif-ali reviewed Oct 14, 2024

View reviewed changes

[juju] Add plugin option for Juju state reporting

f389bfa

Signed-off-by: Michael Thamm <[email protected]>

MichaelThamm force-pushed the report-juju branch from 045cef0 to f389bfa Compare October 15, 2024 14:49

MichaelThamm marked this pull request as ready for review October 15, 2024 15:41

TurboTurtle reviewed Oct 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[juju] Add plugin option for Juju state reporting #3803

[juju] Add plugin option for Juju state reporting #3803

MichaelThamm commented Oct 10, 2024 •

edited

Loading

packit-as-a-service bot commented Oct 10, 2024

arif-ali left a comment

jcastill commented Oct 11, 2024

jcastill commented Oct 13, 2024

MichaelThamm commented Oct 14, 2024

arif-ali commented Oct 14, 2024

MichaelThamm commented Oct 15, 2024

arif-ali commented Oct 15, 2024

MichaelThamm commented Oct 15, 2024

TurboTurtle Oct 15, 2024

MichaelThamm Oct 16, 2024 •

edited

Loading

TurboTurtle Oct 16, 2024

TurboTurtle Oct 15, 2024

TurboTurtle Oct 15, 2024

TurboTurtle Oct 15, 2024

TurboTurtle Oct 15, 2024

[juju] Add plugin option for Juju state reporting #3803

Are you sure you want to change the base?

[juju] Add plugin option for Juju state reporting #3803

Conversation

MichaelThamm commented Oct 10, 2024 • edited Loading

packit-as-a-service bot commented Oct 10, 2024

arif-ali left a comment

Choose a reason for hiding this comment

jcastill commented Oct 11, 2024

jcastill commented Oct 13, 2024

MichaelThamm commented Oct 14, 2024

arif-ali commented Oct 14, 2024

MichaelThamm commented Oct 15, 2024

arif-ali commented Oct 15, 2024

MichaelThamm commented Oct 15, 2024

TurboTurtle Oct 15, 2024

Choose a reason for hiding this comment

MichaelThamm Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

TurboTurtle Oct 16, 2024

Choose a reason for hiding this comment

TurboTurtle Oct 15, 2024

Choose a reason for hiding this comment

TurboTurtle Oct 15, 2024

Choose a reason for hiding this comment

TurboTurtle Oct 15, 2024

Choose a reason for hiding this comment

TurboTurtle Oct 15, 2024

Choose a reason for hiding this comment

MichaelThamm commented Oct 10, 2024 •

edited

Loading

MichaelThamm Oct 16, 2024 •

edited

Loading