Skip to content

Fix ordering for systemd remote journal support

Luis Chamberlain kdevops 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest6 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6mirror: add a smart git checkkconfigs/Kconfig.mirror scripts/test_git_firewall.sh+36 -50 replies
2/6bringup: split provisioning into 2 steps.gitignore Makefile scripts/guestfs.Makefile +2 more+59 -140 replies
3/6bringup: share bringup method targe and agument by two steps.gitignore Makefile scripts/guestfs.Makefile +2 more+70 -910 replies
4/6provision: move all provisioning things to its own MakefileMakefile scripts/provision.Makefile+75 -680 replies
5/6bringup: move journal-server setup earlyMakefile scripts/bringup.Makefile scripts/journal-server.Makefile+57 -560 replies
6/6journal-server: fix by adjusting orderingplaybooks/roles/devconfig/tasks/main.yml scripts/journal-server.Makefile+3 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/6] Fix ordering for systemd remote journal support

For those that may have missed it, in December I added support for systemd remote journals. This allows us to:

1) reduce the workflow watchdog time, since it is now on the localhost
2) always lets us collect node logs in case they crash and the node is gone after so we can't insepct it
3) let's us use journalctl to look for data on the target nodes

The implementation however was to address a series of issues when one is using and enabling SOAK_DURATION of 2.5 hours (so that is CONFIG_FSTESTS_SOAK_DURATION=9900), and so it was designed to cure an existing running system I was fet up with not being able to collect logs for, as crashes are much more frequent to the point you really do often loose access to the system / console.

Since I implemented support for a running kdevops system I simply togggled on CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE=y and ran 'make' and then:

make journal-server
make journal-client

And then use it. The other commands are:

journal-restart - Restart client upload service
journal-status - Ensure systemd-journal-remote works
journal-ls - List journals available and sizes
journal-ln - Add symlinks with hostnames

The issue with this is I had not tested a new cluster bringup. After Daniel Gomez struggled with that, I now looked into that and have identified the issues. The fix is to enhance our semantics on bringup by defining clearly what a provider is and also add clear semantics to allow us to install things *prior* to running the generic devconfig playbook with all the bells and whistles.

The last few patches should demonstrate how easy it is now to add new install targets for things we need in the pipeline for provisioning.

This should bring using CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE closer to a reality for most folks. The only nagging thing I don't like about it is, is that we require sudo on the local system to query the client logs, and that pollutes your systems's logs with a ton of messages with sudo.

So the next thing is to verify we can remove sudo from the watchdog calls, since we already took care of making the systemd remote journal directory with a sticky bit, and then ensured our user is part of the group systemd-journal-remote so to allow our user to read the remote journals without sudo. We just gotta test this a bit more wider.

So other than patch review, it's now a good time to ask for wider testing with CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE=y and so we can default CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE=y later.

Please let me know what you think.

Luis Chamberlain (6):
mirror: add a smart git check
bringup: split provisioning into 2 steps
bringup: share bringup method targe and agument by two steps
provision: move all provisioning things to its own Makefile
bringup: move journal-server setup early
journal-server: fix by adjusting ordering

 .gitignore                               |  2 +-
 Makefile                                 | 31 ++--------
 kconfigs/Kconfig.mirror                  | 24 ++++++--
 playbooks/roles/devconfig/tasks/main.yml |  1 +
 scripts/bringup.Makefile                 | 56 ------------------
 scripts/guestfs.Makefile                 | 19 ++----
 scripts/journal-server.Makefile          | 56 ++++++++++++++++++
 scripts/provision.Makefile               | 74 ++++++++++++++++++++++++
 scripts/terraform.Makefile               | 10 ++--
 scripts/test_git_firewall.sh             | 17 ++++++
 scripts/vagrant.Makefile                 | 19 ++----
 11 files changed, 187 insertions(+), 122 deletions(-)
 create mode 100644 scripts/journal-server.Makefile
 create mode 100644 scripts/provision.Makefile
 create mode 100755 scripts/test_git_firewall.sh
Signature · 1 line
2.42.0