package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in /dev/shm folder and does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].
To make sure we can remove this leftovers of sysrepoctl we specify a build specific SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.
Fixes:
[1] http://autobuild.buildroot.net/results/6e559c4f98b7ed93d7b5af638264e907492a6532/
--- package/netopeer2/netopeer2.mk | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
package/netopeer2/netopeer2.mk +14 -0
@@ -13,9 +13,23 @@ NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF) +# Set a build specific SYSREPO_SHM_PREFIX to ensure we can safely delete the +# files. This also ensures that concurrent parallel builds will not be +# affected mutualy. +SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR)) +NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX) + define NETOPEER2_INSTALL_INIT_SYSV $(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \ $(TARGET_DIR)/etc/init.d/S52netopeer2 endef +# The host sysrepo used to install the netopeer2 modules will leave +# its shared memory files lingering about. Clean up in it's stead... +define NETOPEER2_CLEANUP + rm -f /dev/shm/$(SYSREPO_SHM_PREFIX)* +endef +NETOPEER2_PRE_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP +NETOPEER2_POST_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP + $(eval $(cmake-package))
Signature · 1 line
2.20.1
PATCH 2/2 package/netopeer2: add dependency to host-sysrepo
The sysrepoctl executable from the host-sysrepo package is used to install YANG modules during installation. So add the dependency here. Also make sure we use this executable by setting the make environment variable SYSREPOCTL_EXECUTABLE. Otherwise a system wide installed sysrepoctl would be used that is not what we want.
--- package/netopeer2/netopeer2.mk | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
package/netopeer2/netopeer2.mk +4 -2
@@ -9,7 +9,7 @@ NETOPEER2_SITE = $(call github,CESNET,Netopeer2,v$(NETOPEER2_VERSION)) NETOPEER2_DL_SUBDIR = netopeer2 NETOPEER2_LICENSE = BSD-3-Clause NETOPEER2_LICENSE_FILES = LICENSE -NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo +NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo host-sysrepo NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF)
@@ -17,7 +17,9 @@ NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF) # files. This also ensures that concurrent parallel builds will not be # affected mutualy. SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR)) -NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX) +NETOPEER2_MAKE_ENV = \ + SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX) \ + SYSREPOCTL_EXECUTABLE=$(HOST_DIR)/bin/sysrepoctl define NETOPEER2_INSTALL_INIT_SYSV $(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \
Signature · 1 line
2.20.1
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
In reply to Heiko ThieryHeiko, All,
On 2021-02-06 11:57 +0100, Heiko Thiery spake thusly:
Quoted 25 lines
On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in
/dev/shmfolder and does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].To make sure we can remove this leftovers of sysrepoctl we specify a build specific
SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.Fixes:
[1] http://autobuild.buildroot.net/results/6e559c4f98b7ed93d7b5af638264e907492a6532/Co-Developed-by Yann E. MORIN <[email protected]>Signed-off-by Heiko Thiery <[email protected]>--- package/netopeer2/netopeer2.mk | 14 ++++++++++++++ 1 file changed, 14 insertions(+)package/netopeer2/netopeer2.mk +0
-0
package/netopeer2/netopeer2.mk NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo · line 19
Quoted 7 lines
package/netopeer2/netopeer2.mk +4
-0@@ -13,9 +13,23 @@ NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF) +# Set a build specific SYSREPO_SHM_PREFIX to ensure we can safely delete the +# files. This also ensures that concurrent parallel builds will not be +# affected mutualy. +SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))
It should be named after the current package, ie. NETOPEER2_SHM_PREFIX.
So, I sent this proposal late yesterday, and of course I forgot some corner cases.
If we have two (independent) packages that call the host sysrepo during build/instal, then we will still have the concurrency issue with top-level parallel builds.
So, this prefix must also contain the name of the current package, i.e. probably something along the lines of:
NETOPEER2_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))_netopeer2
Of course, ideally, we would like to make that generic, but this is still too early for now. That would probably imply a macro helper, like:
sysrepo-make-shm-prefix = sr_buildroot$(subst /,_,$(CONFIG_DIR))_$(PKG)
and then packages would use it as:
NETOPEER2_SHM_PREFIX = $(call sysrepo-make-shm-prefix)
(the 'call' is superfluous, but it since we already have a bunch of helpers that have to be called, it is nicer that they all should be.)
But really, that's still too early...
package/netopeer2/netopeer2.mk NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo · line 28
Quoted 9 lines
package/netopeer2/netopeer2.mk +4
-0+NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX) + define NETOPEER2_INSTALL_INIT_SYSV $(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \ $(TARGET_DIR)/etc/init.d/S52netopeer2 endef +# The host sysrepo used to install the netopeer2 modules will leave +# its shared memory files lingering about. Clean up in it's stead...
s/it's/its/
https://dictionary.cambridge.org/grammar/british-grammar/it-s-or-its
When I said I hand-edited the patch, and that it may be broken, that was not the part I was thinking about... ;-)
No need to resend, I'll fix those when applying...
Regards,
Yann E. MORIN.
Quoted 10 lines
package/netopeer2/netopeer2.mk +6
-0+define NETOPEER2_CLEANUP + rm -f /dev/shm/$(SYSREPO_SHM_PREFIX)* +endef +NETOPEER2_PRE_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP +NETOPEER2_POST_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP + $(eval $(cmake-package))Signature · 1 line
2.20.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
In reply to Heiko ThieryHello Heiko, Yann,
Quoted 2 lines · Heiko Thiery
On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in
/dev/shmfolder and
Really install? The make log shows:
-- Installing: .../target/usr/share/yang/modules/netopeer2/notifications at 2008-07-14.yang
is installed already before the setup.sh/sysrepoctl step
Quoted 8 lines
does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].
To make sure we can remove this leftovers of sysrepoctl we specify a build specific
SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.
Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?
The setup.sh is called with the following environment variables:
NP2_MODULE_DIR /usr/share/yang/modules/netopeer2 NP2_MODULE_PERMS 600 NP2_MODULE_OWNER seiderer NP2_MODULE_GROUP users
A (quick) strace check of setup.sh run shows no other output than the shm files...
Regards,
Peter
Quoted 38 lines
Fixes:
[1] http://autobuild.buildroot.net/results/6e559c4f98b7ed93d7b5af638264e907492a6532/Co-Developed-by Yann E. MORIN <[email protected]>Signed-off-by Heiko Thiery <[email protected]>--- package/netopeer2/netopeer2.mk | 14 ++++++++++++++ 1 file changed, 14 insertions(+)package/netopeer2/netopeer2.mk +14
-0@@ -13,9 +13,23 @@ NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF) +# Set a build specific SYSREPO_SHM_PREFIX to ensure we can safely delete the +# files. This also ensures that concurrent parallel builds will not be +# affected mutualy. +SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR)) +NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX) + define NETOPEER2_INSTALL_INIT_SYSV $(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \ $(TARGET_DIR)/etc/init.d/S52netopeer2 endef +# The host sysrepo used to install the netopeer2 modules will leave +# its shared memory files lingering about. Clean up in it's stead... +define NETOPEER2_CLEANUP + rm -f /dev/shm/$(SYSREPO_SHM_PREFIX)* +endef +NETOPEER2_PRE_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP +NETOPEER2_POST_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP + $(eval $(cmake-package))
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
Am So., 7. Feb. 2021 um 12:39 Uhr schrieb Peter Seiderer <[email protected]>:
Quoted 5 lines
Hello Heiko, Yann,
On Sat, 6 Feb 2021 11:57:35 +0100, Heiko Thiery <[email protected]> wrote:
Quoted 8 lines
Quoted 2 lines
On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in
/dev/shmfolder andReally install? The make log shows:
-- Installing: .../target/usr/share/yang/modules/netopeer2/notifications at 2008-07-14.yangis installed already before the setup.sh/sysrepoctl step
By saying install not only the install target that copies the yang files into /usr/share/yang/modules/netopeer2/* is meant. The setup.sh will use "sysrepoctl install" to activate and configure the yang modules in sysrepo. With this the runtime files in /dev/shm/sr_* and the startup files in <TARGET>/etc/sysrepo/data/* files are created. The runtime files will be again created on the target when sysrepo/netopeer2 is started. So the /dev/shm/sr_* files are only temporary.
Quoted 15 lines
Quoted 9 lines
does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].
To make sure we can remove this leftovers of sysrepoctl we specify a build specific
SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?
Quoted 8 lines
The
setup.shis called with the following environment variables:NP2_MODULE_DIR /usr/share/yang/modules/netopeer2 NP2_MODULE_PERMS 600 NP2_MODULE_OWNER seiderer NP2_MODULE_GROUP usersA (quick) strace check of
setup.shrun shows no other output than the shm files...
As far I can see there are files in <target>/etc/sysrepo that should be created by the setup.sh files. So I cannot understand why you do not see these files. Did you do a reinstall or a clean build?
Thank you
Signature · 1 line
Heiko
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
Hello Heiko,
Quoted 46 lines · Heiko Thiery
Am So., 7. Feb. 2021 um 12:39 Uhr schrieb Peter Seiderer <[email protected]>:
Quoted 5 lines
Hello Heiko, Yann,
On Sat, 6 Feb 2021 11:57:35 +0100, Heiko Thiery <[email protected]> wrote:
Quoted 8 lines
Quoted 2 lines
On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in
/dev/shmfolder andReally install? The make log shows:
-- Installing: .../target/usr/share/yang/modules/netopeer2/notifications at 2008-07-14.yangis installed already before the setup.sh/sysrepoctl step
By saying install not only the install target that copies the yang files into
/usr/share/yang/modules/netopeer2/*is meant. Thesetup.shwill use "sysrepoctl install" to activate and configure the yang modules in sysrepo. With this the runtime files in/dev/shm/sr_*and the startup files in <TARGET>/etc/sysrepo/data/* files are created. The runtime files will be again created on the target when sysrepo/netopeer2 is started. So the/dev/shm/sr_*files are only temporary.Quoted 22 lines
Quoted 9 lines
does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].
To make sure we can remove this leftovers of sysrepoctl we specify a build specific
SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?
The
setup.shis called with the following environment variables:NP2_MODULE_DIR /usr/share/yang/modules/netopeer2 NP2_MODULE_PERMS 600 NP2_MODULE_OWNER seiderer NP2_MODULE_GROUP users
Quoted 5 lines
Quoted 1 line
A (quick) strace check of
setup.shrun shows no other output than the shm files...As far I can see there are files in <target>/etc/sysrepo that should be created by the
setup.shfiles. So I cannot understand why you do not see these files. Did you do a reinstall or a clean build?
Yes your are right, did not look close enough and missed this ones;-)
Thanks for clarifying!
Regards,
Peter
Quoted 2 lines
Thank you
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
In reply to Peter SeidererQuoted 7 lines · Peter Seiderer
Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?
The
setup.shis called with the following environment variables:
Quoted 7 lines
NP2_MODULE_DIR /usr/share/yang/modules/netopeer2 NP2_MODULE_PERMS 600 NP2_MODULE_OWNER seiderer NP2_MODULE_GROUP usersA (quick) strace check of
setup.shrun shows no other output than the shm files...
Disclaimer: While I work for the same company as the sysrepo+netopeer2 maintainers, I'm not upstream. On the other hand, we've been using sysrepo+netopeer2 on ARM via Buildroot for a few years. Here's what we're doing. Our use case is having a common buildroot build image for several appliances that each use a slightly different set of YANG modules. We also have a read-only rootfs, with a separate partitions for stateful data, and we're using the A/B boot slots via RAUC.
Our first attempt was persisting the /etc/sysrepo directory across system updates. That did not work well because of some version bumps of internal modules. Also, the current build scripts upstream can cope with partial state in /etc/sysrepo, but that does not help for buildroot because the state at the build time is very different from the state at the boot time. In my experience the errors were not intuitive, and the fact that /etc/sysrepo contains binary LYB files does not help here.
In our current approach, we do not install any YANG modules at build time -- not even for Netopeer2-server. Instead, we're using a series of systemd units:
- The first one prepares a tmpfs for /run/sysrepo that we're using instead of /dev/shm. This is important for us because the SHM code in sysrpeo has been (so far) showing rather serious bugs, and it occasionally helps to just start from scratch without a reboot. We're using also addiing a few stanzas to any service which uses sysrepo: BindsTo=run-sysrepo.mount, After=run-sysrepo.mount, PartOf=run-sysrepo.mount, and a PartOf=$${UNIT} for each of this unit in a drop-in file for the run-sysrepo.mount unit file.
- Then we call netopeer2's shell script for installing the required YANG modules. That's the same shell script which upstream would regularly use at the install time. This only installs some models, there's no configuration.
- Then we call our internal shell script which decides what application-specififc YANG modules should be installed, and installs them. Some of these modules require initial "factory" data (and they would not pass validation if they were left empty), so these are installed as well. It is critical that nothing runs in parallel to these.
- The we restore configuration from the previous reboot, if any. That's a simple sysrepocfg -d startup -f json --import=/cfg/sysrepo/startup.json followed by sysrepocfg -C startup.
- Then we apply the NACM rule set -- we have some systemwide YANG modules and an app that processes them for authentication, FW updates, etc, which have to be ACL-limited.
- The we check if the Netopeer2 server already has a listening endpoint configured via the sysrepo database content. There are essentially three possible scenarios:
1) There's been no persistent configuration to restore from the last
boot. In that case, a new configuration should be provisioned.
2) There's been some configuration that we restored earlier, and it
contains a <listen> endpoint or a <call-home> endpoint. If that's the case, good, we're done here.
3) Otherwise, there's been some config, but the NETCONF service has no
listening endpoints configured. In that case, run the configuration scripts from upstream.
In options 1) and 3), we run upstream's merge_hostkey.sh andmerge_config.sh scripts. These actually generate an SSH server host key, so they cannot possibly run during buildroot build time.
- Everything is configured, so we can start launching various services which use sysrepo.
- Finally, there's a horribly inotify script which snapshots the configuration for the next boot: while true; do inotifywait -e CLOSE_WRITE /etc/sysrepo/data/*.startup && mkdir -p /cfg/sysrepo/ && sysrepocfg -d startup -f json -X > /cfg/sysrepo/startup.json; done
Our buildroot config for Netopeer2 is available here:
https://gerrit.cesnet.cz/plugins/gitiles/github/buildroot/buildroot/+/refs/heads/cesnet/2021-01-11/package/netopeer2/
. The scaffolding around that is, unfortunately, not public at this point, but I'll see if I can contribute the key parts over the next months (realistically, April is the soonest one).
Now, to the question asked in this thread about what "installing a YANG file" actually performs in sysrepo, and what the SHM files are. The SHM files serve two purposes, hte first one is synchronizing access when multiple processes are accessing sysrepo. In the past sysrpeo versions, this would be the sysrepod process which is long gone. The second purpose of the SHM files are keeping track of data in the "running datastore". That's a term from the YANG/NETCONF/RESTCONF/NMDA series of RFCs, the TL;DR version is that when the system boots, the content of the persistent configuration is copied to the "running configuration". Often, the system would react to changes in "running" imemdiately, while changes to "startup" are only applied later, but it can be more complex, see RFC 8342 and enjoy the list of optional features and conditional behavior if you're bored.
Based on our experience, we've decided that installing YANG modules at build time is *not* a feasible path forward. If you insist on getting this working, then I strongly suggest using a single location outside of build/ and per-package/ directories -- introduce a new one, build host-sysrepo with the repo path pointing there, and in the rootfs-finalize hook, copy its content to target/etc/sysrepo. There will still be races, and having per-package SHM directories would open a whole new can of worms.
The drawbacks of our current approach is that we're matinaining a single shell script that contains a list of modules to install, and these modules often belong to different packages, so it's inelegant. I can imagine improving this via agreeing on a single location where packages would drop their installation scripts in. Still, to support the R/O rootfs + perisstent-config-somewhere-else scenario, one will need at least two directories. It might work like this:
- /usr/share/sysrepo/install-yang/*.sh , these files provided by packages invoke sysrepoctl and sysrepocfg to install various YANG files, enable YANG-level features in them, and provide the initial "factory" data
- /usr/share/sysrepo/configure-yang/*.sh, these hooks are once again provided by packages, and they perorm some sanity checks to make sure there's a viable configuration. This is where the netopeer2-server would create its SSH key. Another candidate are NACM rules from extra packages.
In between these two hooks that's the time where the configuration could be restored for those users like me who have a r/o rootfs.
Hope this helps at least a bit.
With kind regards,
Jan
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
Jan, All,
On 2021-02-11 14:56 +0100, Jan Kundr?t spake thusly:
Quoted 16 lines · Peter Seiderer
Quoted 7 lines
Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?
The
setup.shis called with the following environment variables:Quoted 7 lines
NP2_MODULE_DIR /usr/share/yang/modules/netopeer2 NP2_MODULE_PERMS 600 NP2_MODULE_OWNER seiderer NP2_MODULE_GROUP usersA (quick) strace check of
setup.shrun shows no other output than the shm files...
Quoted 7 lines
Disclaimer: While I work for the same company as the sysrepo+netopeer2 maintainers, I'm not upstream. On the other hand, we've been using sysrepo+netopeer2 on ARM via Buildroot for a few years. Here's what we're doing. Our use case is having a common buildroot build image for several appliances that each use a slightly different set of YANG modules. We also have a read-only rootfs, with a separate partitions for stateful data, and we're using the A/B boot slots via RAUC.
Thank you for these extensive explanations; very interesting! ?
I eventually decided to apply the patch from Heiko, because it does fix an issue we have currently.
Also, even though your experience suggests not installing things at build time, but only at run time (and I have no argument against that), I believe Heiko is actually using netopeer2, and that the way it is packaged in Buildroot at least works for him, in his very particular case.
So, even if what we have now is not perfect, it seems to work well-enough to be usable, especially now that we can avoid conflicts when installing the YANG modules at build/install time.
Again, thank you very much for your extensive and very valuable feedback!
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
In reply to Jan KundrátHi Jan,
Am Do., 11. Feb. 2021 um 14:56 Uhr schrieb Jan Kundr?t <[email protected]>:
Quoted 91 lines · Peter Seiderer
Quoted 7 lines
Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?
The
setup.shis called with the following environment variables:Quoted 7 lines
NP2_MODULE_DIR /usr/share/yang/modules/netopeer2 NP2_MODULE_PERMS 600 NP2_MODULE_OWNER seiderer NP2_MODULE_GROUP usersA (quick) strace check of
setup.shrun shows no other output than the shm files...Disclaimer: While I work for the same company as the sysrepo+netopeer2 maintainers, I'm not upstream. On the other hand, we've been using sysrepo+netopeer2 on ARM via Buildroot for a few years. Here's what we're doing. Our use case is having a common buildroot build image for several appliances that each use a slightly different set of YANG modules. We also have a read-only rootfs, with a separate partitions for stateful data, and we're using the A/B boot slots via RAUC.
Our first attempt was persisting the
/etc/sysrepodirectory across system updates. That did not work well because of some version bumps of internal modules. Also, the current build scripts upstream can cope with partial state in/etc/sysrepo, but that does not help for buildroot because the state at the build time is very different from the state at the boot time. In my experience the errors were not intuitive, and the fact that/etc/sysrepocontains binary LYB files does not help here.In our current approach, we do not install any YANG modules at build time -- not even for Netopeer2-server. Instead, we're using a series of systemd units:
- The first one prepares a tmpfs for
/run/sysrepothat we're using instead of/dev/shm. This is important for us because the SHM code in sysrpeo has been (so far) showing rather serious bugs, and it occasionally helps to just start from scratch without a reboot. We're using also addiing a few stanzas to any service which uses sysrepo: BindsTo=run-sysrepo.mount, After=run-sysrepo.mount, PartOf=run-sysrepo.mount, and a PartOf=$${UNIT} for each of this unit in a drop-in file for the run-sysrepo.mount unit file.- Then we call netopeer2's shell script for installing the required YANG modules. That's the same shell script which upstream would regularly use at the install time. This only installs some models, there's no configuration.
- Then we call our internal shell script which decides what application-specififc YANG modules should be installed, and installs them. Some of these modules require initial "factory" data (and they would not pass validation if they were left empty), so these are installed as well. It is critical that nothing runs in parallel to these.
- The we restore configuration from the previous reboot, if any. That's a simple
sysrepocfg -d startup -f json --import=/cfg/sysrepo/startup.jsonfollowed bysysrepocfg -C startup.- Then we apply the NACM rule set -- we have some systemwide YANG modules and an app that processes them for authentication, FW updates, etc, which have to be ACL-limited.
- The we check if the Netopeer2 server already has a listening endpoint configured via the sysrepo database content. There are essentially three possible scenarios:
1) There's been no persistent configuration to restore from the last
boot. In that case, a new configuration should be provisioned.2) There's been some configuration that we restored earlier, and it
contains a <listen> endpoint or a <call-home> endpoint. If that's the case, good, we're done here.3) Otherwise, there's been some config, but the NETCONF service has no
listening endpoints configured. In that case, run the configuration scripts from upstream.In options 1) and 3), we run upstream's
merge_hostkey.shandmerge_config.shscripts. These actually generate an SSH server host key, so they cannot possibly run during buildroot build time.- Everything is configured, so we can start launching various services which use sysrepo.
- Finally, there's a horribly inotify script which snapshots the configuration for the next boot:
while true; do inotifywait -e CLOSE_WRITE /etc/sysrepo/data/*.startup && mkdir -p /cfg/sysrepo/ && sysrepocfg -d startup -f json -X > /cfg/sysrepo/startup.json; done
Quoted 5 lines
Our buildroot config for Netopeer2 is available here:
https://gerrit.cesnet.cz/plugins/gitiles/github/buildroot/buildroot/+/refs/heads/cesnet/2021-01-11/package/netopeer2/
. The scaffolding around that is, unfortunately, not public at this point, but I'll see if I can contribute the key parts over the next months (realistically, April is the soonest one).
Sounds great!
Quoted 38 lines
Now, to the question asked in this thread about what "installing a YANG file" actually performs in sysrepo, and what the SHM files are. The SHM files serve two purposes, hte first one is synchronizing access when multiple processes are accessing sysrepo. In the past sysrpeo versions, this would be the
sysrepodprocess which is long gone. The second purpose of the SHM files are keeping track of data in the "running datastore". That's a term from the YANG/NETCONF/RESTCONF/NMDA series of RFCs, the TL;DR version is that when the system boots, the content of the persistent configuration is copied to the "running configuration". Often, the system would react to changes in "running" imemdiately, while changes to "startup" are only applied later, but it can be more complex, see RFC 8342 and enjoy the list of optional features and conditional behavior if you're bored.Based on our experience, we've decided that installing YANG modules at build time is *not* a feasible path forward. If you insist on getting this working, then I strongly suggest using a single location outside of build/ and per-package/ directories -- introduce a new one, build host-sysrepo with the repo path pointing there, and in the rootfs-finalize hook, copy its content to target/etc/sysrepo. There will still be races, and having per-package SHM directories would open a whole new can of worms.
The drawbacks of our current approach is that we're matinaining a single shell script that contains a list of modules to install, and these modules often belong to different packages, so it's inelegant. I can imagine improving this via agreeing on a single location where packages would drop their installation scripts in. Still, to support the R/O rootfs + perisstent-config-somewhere-else scenario, one will need at least two directories. It might work like this:
-
/usr/share/sysrepo/install-yang/*.sh, these files provided by packages invokesysrepoctlandsysrepocfgto install various YANG files, enable YANG-level features in them, and provide the initial "factory" data-
/usr/share/sysrepo/configure-yang/*.sh, these hooks are once again provided by packages, and they perorm some sanity checks to make sure there's a viable configuration. This is where the netopeer2-server would create its SSH key. Another candidate are NACM rules from extra packages.
Quoted 4 lines
In between these two hooks that's the time where the configuration could be restored for those users like me who have a r/o rootfs.
Hope this helps at least a bit.
Thank you for this detailed explanation. I have to admit that I use the packages for sysrepo/netopeer2 in a not that mature fashion like you.
I understand your concerns not to do the installation at build time. But without that currently it is not that easy to get a working installation and it would be hard for users of this package to get it up and running. So my idea is to have the module installation optional in the first step. With that we can switch between having an out of the box working netopeer2 installation and the possibility to do the configuration/installation at startup like you explained above.
Thank you very much.
Signature · 1 line
Heiko
PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
In reply to Heiko ThieryHeiko, All,
On 2021-02-06 11:57 +0100, Heiko Thiery spake thusly:
Quoted 11 lines
On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in
/dev/shmfolder and does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].To make sure we can remove this leftovers of sysrepoctl we specify a build specific
SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.
Quoted 5 lines
Fixes:
[1] http://autobuild.buildroot.net/results/6e559c4f98b7ed93d7b5af638264e907492a6532/Co-Developed-by Yann E. MORIN <[email protected]>Signed-off-by Heiko Thiery <[email protected]>
Series of two applied to master, thanks.
I inverted the order in which the two patches are applied, because I found it to be more logical: we need host-sysrepo, and then we can fix its usage...
As for the now-second patch, I did what I said in my previous review, plus a few litle other cleanuos and additions:
- also use the package name as discriminant
- expand commit log accordingly
- rename the variable to start with the package name
- explain why we clean up before as well
Thanks! :-)
Regards,
Yann E. MORIN.
Quoted 39 lines
--- package/netopeer2/netopeer2.mk | 14 ++++++++++++++ 1 file changed, 14 insertions(+)package/netopeer2/netopeer2.mk +14
-0@@ -13,9 +13,23 @@ NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF) +# Set a build specific SYSREPO_SHM_PREFIX to ensure we can safely delete the +# files. This also ensures that concurrent parallel builds will not be +# affected mutualy. +SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR)) +NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX) + define NETOPEER2_INSTALL_INIT_SYSV $(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \ $(TARGET_DIR)/etc/init.d/S52netopeer2 endef +# The host sysrepo used to install the netopeer2 modules will leave +# its shared memory files lingering about. Clean up in it's stead... +define NETOPEER2_CLEANUP + rm -f /dev/shm/$(SYSREPO_SHM_PREFIX)* +endef +NETOPEER2_PRE_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP +NETOPEER2_POST_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP + $(eval $(cmake-package))Signature · 6 lines
2.20.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'