Skip to contentSkip to conversation

package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

PATCH New Heiko Thiery 4 participants 10 messages Feb 6, 2021 – Feb 11, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in /dev/shm folder and does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].

To make sure we can remove this leftovers of sysrepoctl we specify a build specific SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.

Fixes:
[1] http://autobuild.buildroot.net/results/6e559c4f98b7ed93d7b5af638264e907492a6532/

---
 package/netopeer2/netopeer2.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
package/netopeer2/netopeer2.mk +14 -0
index bc02e0dc93..af30d8417a 100644
@@ -13,9 +13,23 @@ NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo
 
 NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF)
 
+# Set a build specific SYSREPO_SHM_PREFIX to ensure we can safely delete the
+# files. This also ensures that concurrent parallel builds will not be
+# affected mutualy.
+SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))
+NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX)
+
 define NETOPEER2_INSTALL_INIT_SYSV
 	$(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \
 		$(TARGET_DIR)/etc/init.d/S52netopeer2
 endef
 
+# The host sysrepo used to install the netopeer2 modules will leave
+# its shared memory files lingering about. Clean up in it's stead...
+define NETOPEER2_CLEANUP
+   rm -f /dev/shm/$(SYSREPO_SHM_PREFIX)*
+endef
+NETOPEER2_PRE_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP
+NETOPEER2_POST_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP
+
 $(eval $(cmake-package))
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/2 package/netopeer2: add dependency to host-sysrepo

The sysrepoctl executable from the host-sysrepo package is used to install YANG modules during installation. So add the dependency here. Also make sure we use this executable by setting the make environment variable SYSREPOCTL_EXECUTABLE. Otherwise a system wide installed sysrepoctl would be used that is not what we want.

---
 package/netopeer2/netopeer2.mk | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
package/netopeer2/netopeer2.mk +4 -2
index af30d8417a..a6022db155 100644
@@ -9,7 +9,7 @@ NETOPEER2_SITE = $(call github,CESNET,Netopeer2,v$(NETOPEER2_VERSION))
 NETOPEER2_DL_SUBDIR = netopeer2
 NETOPEER2_LICENSE = BSD-3-Clause
 NETOPEER2_LICENSE_FILES = LICENSE
-NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo
+NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo host-sysrepo
 
 NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF)
 
@@ -17,7 +17,9 @@ NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF)
 # files. This also ensures that concurrent parallel builds will not be
 # affected mutualy.
 SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))
-NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX)
+NETOPEER2_MAKE_ENV = \
+	SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX) \
+	SYSREPOCTL_EXECUTABLE=$(HOST_DIR)/bin/sysrepoctl
 
 define NETOPEER2_INSTALL_INIT_SYSV
 	$(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \
Signature · 1 line
2.20.1
#2 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

In reply to Heiko Thiery

Heiko, All,

On 2021-02-06 11:57 +0100, Heiko Thiery spake thusly:

Quoted 25 lines

On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in /dev/shm folder and does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].

To make sure we can remove this leftovers of sysrepoctl we specify a build specific SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.

Fixes:
[1] http://autobuild.buildroot.net/results/6e559c4f98b7ed93d7b5af638264e907492a6532/

---
 package/netopeer2/netopeer2.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
package/netopeer2/netopeer2.mk +0 -0
index bc02e0dc93..af30d8417a 100644

package/netopeer2/netopeer2.mk NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo · line 19

Quoted 7 lines
package/netopeer2/netopeer2.mk +4 -0
@@ -13,9 +13,23 @@ NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo
 
 NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF)
 
+# Set a build specific SYSREPO_SHM_PREFIX to ensure we can safely delete the
+# files. This also ensures that concurrent parallel builds will not be
+# affected mutualy.
+SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))

It should be named after the current package, ie. NETOPEER2_SHM_PREFIX.

So, I sent this proposal late yesterday, and of course I forgot some corner cases.

If we have two (independent) packages that call the host sysrepo during build/instal, then we will still have the concurrency issue with top-level parallel builds.

So, this prefix must also contain the name of the current package, i.e. probably something along the lines of:

    NETOPEER2_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))_netopeer2

Of course, ideally, we would like to make that generic, but this is still too early for now. That would probably imply a macro helper, like:

    sysrepo-make-shm-prefix = sr_buildroot$(subst /,_,$(CONFIG_DIR))_$(PKG)

and then packages would use it as:

    NETOPEER2_SHM_PREFIX = $(call sysrepo-make-shm-prefix)

(the 'call' is superfluous, but it since we already have a bunch of helpers that have to be called, it is nicer that they all should be.)

But really, that's still too early...

package/netopeer2/netopeer2.mk NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo · line 28

Quoted 9 lines
package/netopeer2/netopeer2.mk +4 -0
+NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX)
+
 define NETOPEER2_INSTALL_INIT_SYSV
 	$(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \
 		$(TARGET_DIR)/etc/init.d/S52netopeer2
 endef
 
+# The host sysrepo used to install the netopeer2 modules will leave
+# its shared memory files lingering about. Clean up in it's stead...

s/it's/its/

https://dictionary.cambridge.org/grammar/british-grammar/it-s-or-its

When I said I hand-edited the patch, and that it may be broken, that was not the part I was thinking about... ;-)

No need to resend, I'll fix those when applying...

Regards,
Yann E. MORIN.

Quoted 10 lines
package/netopeer2/netopeer2.mk +6 -0
+define NETOPEER2_CLEANUP
+   rm -f /dev/shm/$(SYSREPO_SHM_PREFIX)*
+endef
+NETOPEER2_PRE_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP
+NETOPEER2_POST_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP
+
 $(eval $(cmake-package))
Signature · 1 line
2.20.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

In reply to Heiko Thiery

Hello Heiko, Yann,

Quoted 2 lines · Heiko Thiery

On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in /dev/shm folder and

Really install? The make log shows:

	-- Installing: .../target/usr/share/yang/modules/netopeer2/notifications at 2008-07-14.yang

is installed already before the setup.sh/sysrepoctl step

Quoted 8 lines

does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].

To make sure we can remove this leftovers of sysrepoctl we specify a build specific SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.

Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?

The setup.sh is called with the following environment variables:

	NP2_MODULE_DIR /usr/share/yang/modules/netopeer2
	NP2_MODULE_PERMS 600
	NP2_MODULE_OWNER seiderer
	NP2_MODULE_GROUP users

A (quick) strace check of setup.sh run shows no other output than the shm files...

Regards,
Peter

Quoted 38 lines

Fixes:
[1] http://autobuild.buildroot.net/results/6e559c4f98b7ed93d7b5af638264e907492a6532/

---
 package/netopeer2/netopeer2.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
package/netopeer2/netopeer2.mk +14 -0
index bc02e0dc93..af30d8417a 100644
@@ -13,9 +13,23 @@ NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo
 
 NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF)
 
+# Set a build specific SYSREPO_SHM_PREFIX to ensure we can safely delete the
+# files. This also ensures that concurrent parallel builds will not be
+# affected mutualy.
+SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))
+NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX)
+
 define NETOPEER2_INSTALL_INIT_SYSV
 	$(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \
 		$(TARGET_DIR)/etc/init.d/S52netopeer2
 endef
 
+# The host sysrepo used to install the netopeer2 modules will leave
+# its shared memory files lingering about. Clean up in it's stead...
+define NETOPEER2_CLEANUP
+   rm -f /dev/shm/$(SYSREPO_SHM_PREFIX)*
+endef
+NETOPEER2_PRE_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP
+NETOPEER2_POST_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP
+
 $(eval $(cmake-package))
#4 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

Am So., 7. Feb. 2021 um 12:39 Uhr schrieb Peter Seiderer <[email protected]>:

Quoted 5 lines

Hello Heiko, Yann,

On Sat, 6 Feb 2021 11:57:35 +0100, Heiko Thiery <[email protected]> wrote:

Quoted 8 lines
Quoted 2 lines

On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in /dev/shm folder and

Really install? The make log shows:

        -- Installing: .../target/usr/share/yang/modules/netopeer2/notifications at 2008-07-14.yang

is installed already before the setup.sh/sysrepoctl step

By saying install not only the install target that copies the yang files into /usr/share/yang/modules/netopeer2/* is meant. The setup.sh will use "sysrepoctl install" to activate and configure the yang modules in sysrepo. With this the runtime files in /dev/shm/sr_* and the startup files in <TARGET>/etc/sysrepo/data/* files are created. The runtime files will be again created on the target when sysrepo/netopeer2 is started. So the /dev/shm/sr_* files are only temporary.

Quoted 15 lines
Quoted 9 lines

does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].

To make sure we can remove this leftovers of sysrepoctl we specify a build specific SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.

Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?

Quoted 8 lines

The setup.sh is called with the following environment variables:

        NP2_MODULE_DIR /usr/share/yang/modules/netopeer2
        NP2_MODULE_PERMS 600
        NP2_MODULE_OWNER seiderer
        NP2_MODULE_GROUP users

A (quick) strace check of setup.sh run shows no other output than the shm files...

As far I can see there are files in <target>/etc/sysrepo that should be created by the setup.sh files. So I cannot understand why you do not see these files. Did you do a reinstall or a clean build?

Thank you

Signature · 1 line
Heiko
#5 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

Hello Heiko,

Quoted 46 lines · Heiko Thiery

Am So., 7. Feb. 2021 um 12:39 Uhr schrieb Peter Seiderer <[email protected]>:

Quoted 5 lines

Hello Heiko, Yann,

On Sat, 6 Feb 2021 11:57:35 +0100, Heiko Thiery <[email protected]> wrote:

Quoted 8 lines
Quoted 2 lines

On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in /dev/shm folder and

Really install? The make log shows:

        -- Installing: .../target/usr/share/yang/modules/netopeer2/notifications at 2008-07-14.yang

is installed already before the setup.sh/sysrepoctl step

By saying install not only the install target that copies the yang files into /usr/share/yang/modules/netopeer2/* is meant. The setup.sh will use "sysrepoctl install" to activate and configure the yang modules in sysrepo. With this the runtime files in /dev/shm/sr_* and the startup files in <TARGET>/etc/sysrepo/data/* files are created. The runtime files will be again created on the target when sysrepo/netopeer2 is started. So the /dev/shm/sr_* files are only temporary.

Quoted 22 lines
Quoted 9 lines

does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].

To make sure we can remove this leftovers of sysrepoctl we specify a build specific SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.

Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?

The setup.sh is called with the following environment variables:

        NP2_MODULE_DIR /usr/share/yang/modules/netopeer2
        NP2_MODULE_PERMS 600
        NP2_MODULE_OWNER seiderer
        NP2_MODULE_GROUP users
Quoted 5 lines
Quoted 1 line

A (quick) strace check of setup.sh run shows no other output than the shm files...

As far I can see there are files in <target>/etc/sysrepo that should be created by the setup.sh files. So I cannot understand why you do not see these files. Did you do a reinstall or a clean build?

Yes your are right, did not look close enough and missed this ones;-)

Thanks for clarifying!

Regards,
Peter

Quoted 2 lines

Thank you

#6 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

In reply to Peter Seiderer
Quoted 7 lines · Peter Seiderer

Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?

The setup.sh is called with the following environment variables:

Quoted 7 lines
	NP2_MODULE_DIR /usr/share/yang/modules/netopeer2
	NP2_MODULE_PERMS 600
	NP2_MODULE_OWNER seiderer
	NP2_MODULE_GROUP users

A (quick) strace check of setup.sh run shows no other output than the shm files...

Disclaimer: While I work for the same company as the sysrepo+netopeer2 maintainers, I'm not upstream. On the other hand, we've been using sysrepo+netopeer2 on ARM via Buildroot for a few years. Here's what we're doing. Our use case is having a common buildroot build image for several appliances that each use a slightly different set of YANG modules. We also have a read-only rootfs, with a separate partitions for stateful data, and we're using the A/B boot slots via RAUC.

Our first attempt was persisting the /etc/sysrepo directory across system updates. That did not work well because of some version bumps of internal modules. Also, the current build scripts upstream can cope with partial state in /etc/sysrepo, but that does not help for buildroot because the state at the build time is very different from the state at the boot time. In my experience the errors were not intuitive, and the fact that /etc/sysrepo contains binary LYB files does not help here.

In our current approach, we do not install any YANG modules at build time -- not even for Netopeer2-server. Instead, we're using a series of systemd units:

- The first one prepares a tmpfs for /run/sysrepo that we're using instead of /dev/shm. This is important for us because the SHM code in sysrpeo has been (so far) showing rather serious bugs, and it occasionally helps to just start from scratch without a reboot. We're using also addiing a few stanzas to any service which uses sysrepo: BindsTo=run-sysrepo.mount, After=run-sysrepo.mount, PartOf=run-sysrepo.mount, and a PartOf=$${UNIT} for each of this unit in a drop-in file for the run-sysrepo.mount unit file.

- Then we call netopeer2's shell script for installing the required YANG modules. That's the same shell script which upstream would regularly use at the install time. This only installs some models, there's no configuration.

- Then we call our internal shell script which decides what application-specififc YANG modules should be installed, and installs them. Some of these modules require initial "factory" data (and they would not pass validation if they were left empty), so these are installed as well. It is critical that nothing runs in parallel to these.

- The we restore configuration from the previous reboot, if any. That's a simple sysrepocfg -d startup -f json --import=/cfg/sysrepo/startup.json followed by sysrepocfg -C startup.

- Then we apply the NACM rule set -- we have some systemwide YANG modules and an app that processes them for authentication, FW updates, etc, which have to be ACL-limited.

- The we check if the Netopeer2 server already has a listening endpoint configured via the sysrepo database content. There are essentially three possible scenarios:

1) There's been no persistent configuration to restore from the last
boot. In that case, a new configuration should be provisioned.

2) There's been some configuration that we restored earlier, and it
contains a <listen> endpoint or a <call-home> endpoint. If that's the case, good, we're done here.

3) Otherwise, there's been some config, but the NETCONF service has no
listening endpoints configured. In that case, run the configuration scripts from upstream.

In options 1) and 3), we run upstream's merge_hostkey.sh and
merge_config.sh scripts. These actually generate an SSH server host key, so they cannot possibly run during buildroot build time.

- Everything is configured, so we can start launching various services which use sysrepo.

- Finally, there's a horribly inotify script which snapshots the configuration for the next boot: while true; do inotifywait -e CLOSE_WRITE /etc/sysrepo/data/*.startup && mkdir -p /cfg/sysrepo/ && sysrepocfg -d startup -f json -X > /cfg/sysrepo/startup.json; done

Our buildroot config for Netopeer2 is available here:
https://gerrit.cesnet.cz/plugins/gitiles/github/buildroot/buildroot/+/refs/heads/cesnet/2021-01-11/package/netopeer2/
. The scaffolding around that is, unfortunately, not public at this point, but I'll see if I can contribute the key parts over the next months (realistically, April is the soonest one).

Now, to the question asked in this thread about what "installing a YANG file" actually performs in sysrepo, and what the SHM files are. The SHM files serve two purposes, hte first one is synchronizing access when multiple processes are accessing sysrepo. In the past sysrpeo versions, this would be the sysrepod process which is long gone. The second purpose of the SHM files are keeping track of data in the "running datastore". That's a term from the YANG/NETCONF/RESTCONF/NMDA series of RFCs, the TL;DR version is that when the system boots, the content of the persistent configuration is copied to the "running configuration". Often, the system would react to changes in "running" imemdiately, while changes to "startup" are only applied later, but it can be more complex, see RFC 8342 and enjoy the list of optional features and conditional behavior if you're bored.

Based on our experience, we've decided that installing YANG modules at build time is *not* a feasible path forward. If you insist on getting this working, then I strongly suggest using a single location outside of build/ and per-package/ directories -- introduce a new one, build host-sysrepo with the repo path pointing there, and in the rootfs-finalize hook, copy its content to target/etc/sysrepo. There will still be races, and having per-package SHM directories would open a whole new can of worms.

The drawbacks of our current approach is that we're matinaining a single shell script that contains a list of modules to install, and these modules often belong to different packages, so it's inelegant. I can imagine improving this via agreeing on a single location where packages would drop their installation scripts in. Still, to support the R/O rootfs + perisstent-config-somewhere-else scenario, one will need at least two directories. It might work like this:

- /usr/share/sysrepo/install-yang/*.sh , these files provided by packages invoke sysrepoctl and sysrepocfg to install various YANG files, enable YANG-level features in them, and provide the initial "factory" data

- /usr/share/sysrepo/configure-yang/*.sh, these hooks are once again provided by packages, and they perorm some sanity checks to make sure there's a viable configuration. This is where the netopeer2-server would create its SSH key. Another candidate are NACM rules from extra packages.

In between these two hooks that's the time where the configuration could be restored for those users like me who have a r/o rootfs.

Hope this helps at least a bit.

With kind regards,
Jan

#7 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

Jan, All,

On 2021-02-11 14:56 +0100, Jan Kundr?t spake thusly:

Quoted 16 lines · Peter Seiderer
Quoted 7 lines

Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?

The setup.sh is called with the following environment variables:

Quoted 7 lines
	NP2_MODULE_DIR /usr/share/yang/modules/netopeer2
	NP2_MODULE_PERMS 600
	NP2_MODULE_OWNER seiderer
	NP2_MODULE_GROUP users

A (quick) strace check of setup.sh run shows no other output than the shm files...

Quoted 7 lines

Disclaimer: While I work for the same company as the sysrepo+netopeer2 maintainers, I'm not upstream. On the other hand, we've been using sysrepo+netopeer2 on ARM via Buildroot for a few years. Here's what we're doing. Our use case is having a common buildroot build image for several appliances that each use a slightly different set of YANG modules. We also have a read-only rootfs, with a separate partitions for stateful data, and we're using the A/B boot slots via RAUC.

Thank you for these extensive explanations; very interesting! ?

I eventually decided to apply the patch from Heiko, because it does fix an issue we have currently.

Also, even though your experience suggests not installing things at build time, but only at run time (and I have no argument against that), I believe Heiko is actually using netopeer2, and that the way it is packaged in Buildroot at least works for him, in his very particular case.

So, even if what we have now is not perfect, it seems to work well-enough to be usable, especially now that we can avoid conflicts when installing the YANG modules at build/install time.

Again, thank you very much for your extensive and very valuable feedback!

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#8 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

In reply to Jan Kundrát

Hi Jan,

Am Do., 11. Feb. 2021 um 14:56 Uhr schrieb Jan Kundr?t <[email protected]>:

Quoted 91 lines · Peter Seiderer
Quoted 7 lines

Still the question, are the shm files needed at runtime? If so they should belong in the target directory (or created by a startup script), of not why create them at first (why not skip the setup.sh/sysrepoctl step)? Are there any other products/output of the setup.sh/sysrepoctl step?

The setup.sh is called with the following environment variables:

Quoted 7 lines
      NP2_MODULE_DIR /usr/share/yang/modules/netopeer2
      NP2_MODULE_PERMS 600
      NP2_MODULE_OWNER seiderer
      NP2_MODULE_GROUP users

A (quick) strace check of setup.sh run shows no other output than the shm files...

Disclaimer: While I work for the same company as the sysrepo+netopeer2 maintainers, I'm not upstream. On the other hand, we've been using sysrepo+netopeer2 on ARM via Buildroot for a few years. Here's what we're doing. Our use case is having a common buildroot build image for several appliances that each use a slightly different set of YANG modules. We also have a read-only rootfs, with a separate partitions for stateful data, and we're using the A/B boot slots via RAUC.

Our first attempt was persisting the /etc/sysrepo directory across system updates. That did not work well because of some version bumps of internal modules. Also, the current build scripts upstream can cope with partial state in /etc/sysrepo, but that does not help for buildroot because the state at the build time is very different from the state at the boot time. In my experience the errors were not intuitive, and the fact that /etc/sysrepo contains binary LYB files does not help here.

In our current approach, we do not install any YANG modules at build time -- not even for Netopeer2-server. Instead, we're using a series of systemd units:

- The first one prepares a tmpfs for /run/sysrepo that we're using instead of /dev/shm. This is important for us because the SHM code in sysrpeo has been (so far) showing rather serious bugs, and it occasionally helps to just start from scratch without a reboot. We're using also addiing a few stanzas to any service which uses sysrepo: BindsTo=run-sysrepo.mount, After=run-sysrepo.mount, PartOf=run-sysrepo.mount, and a PartOf=$${UNIT} for each of this unit in a drop-in file for the run-sysrepo.mount unit file.

- Then we call netopeer2's shell script for installing the required YANG modules. That's the same shell script which upstream would regularly use at the install time. This only installs some models, there's no configuration.

- Then we call our internal shell script which decides what application-specififc YANG modules should be installed, and installs them. Some of these modules require initial "factory" data (and they would not pass validation if they were left empty), so these are installed as well. It is critical that nothing runs in parallel to these.

- The we restore configuration from the previous reboot, if any. That's a simple sysrepocfg -d startup -f json --import=/cfg/sysrepo/startup.json followed by sysrepocfg -C startup.

- Then we apply the NACM rule set -- we have some systemwide YANG modules and an app that processes them for authentication, FW updates, etc, which have to be ACL-limited.

- The we check if the Netopeer2 server already has a listening endpoint configured via the sysrepo database content. There are essentially three possible scenarios:

1) There's been no persistent configuration to restore from the last
boot. In that case, a new configuration should be provisioned.

2) There's been some configuration that we restored earlier, and it
contains a <listen> endpoint or a <call-home> endpoint. If that's the case, good, we're done here.

3) Otherwise, there's been some config, but the NETCONF service has no
listening endpoints configured. In that case, run the configuration scripts from upstream.

In options 1) and 3), we run upstream's merge_hostkey.sh and
merge_config.sh scripts. These actually generate an SSH server host key, so they cannot possibly run during buildroot build time.

- Everything is configured, so we can start launching various services which use sysrepo.

- Finally, there's a horribly inotify script which snapshots the configuration for the next boot: while true; do inotifywait -e CLOSE_WRITE /etc/sysrepo/data/*.startup && mkdir -p /cfg/sysrepo/ && sysrepocfg -d startup -f json -X > /cfg/sysrepo/startup.json; done

Quoted 5 lines

Our buildroot config for Netopeer2 is available here:
https://gerrit.cesnet.cz/plugins/gitiles/github/buildroot/buildroot/+/refs/heads/cesnet/2021-01-11/package/netopeer2/
. The scaffolding around that is, unfortunately, not public at this point, but I'll see if I can contribute the key parts over the next months (realistically, April is the soonest one).

Sounds great!

Quoted 38 lines

Now, to the question asked in this thread about what "installing a YANG file" actually performs in sysrepo, and what the SHM files are. The SHM files serve two purposes, hte first one is synchronizing access when multiple processes are accessing sysrepo. In the past sysrpeo versions, this would be the sysrepod process which is long gone. The second purpose of the SHM files are keeping track of data in the "running datastore". That's a term from the YANG/NETCONF/RESTCONF/NMDA series of RFCs, the TL;DR version is that when the system boots, the content of the persistent configuration is copied to the "running configuration". Often, the system would react to changes in "running" imemdiately, while changes to "startup" are only applied later, but it can be more complex, see RFC 8342 and enjoy the list of optional features and conditional behavior if you're bored.

Based on our experience, we've decided that installing YANG modules at build time is *not* a feasible path forward. If you insist on getting this working, then I strongly suggest using a single location outside of build/ and per-package/ directories -- introduce a new one, build host-sysrepo with the repo path pointing there, and in the rootfs-finalize hook, copy its content to target/etc/sysrepo. There will still be races, and having per-package SHM directories would open a whole new can of worms.

The drawbacks of our current approach is that we're matinaining a single shell script that contains a list of modules to install, and these modules often belong to different packages, so it's inelegant. I can imagine improving this via agreeing on a single location where packages would drop their installation scripts in. Still, to support the R/O rootfs + perisstent-config-somewhere-else scenario, one will need at least two directories. It might work like this:

- /usr/share/sysrepo/install-yang/*.sh , these files provided by packages invoke sysrepoctl and sysrepocfg to install various YANG files, enable YANG-level features in them, and provide the initial "factory" data

- /usr/share/sysrepo/configure-yang/*.sh, these hooks are once again provided by packages, and they perorm some sanity checks to make sure there's a viable configuration. This is where the netopeer2-server would create its SSH key. Another candidate are NACM rules from extra packages.

Quoted 4 lines

In between these two hooks that's the time where the configuration could be restored for those users like me who have a r/o rootfs.

Hope this helps at least a bit.

Thank you for this detailed explanation. I have to admit that I use the packages for sysrepo/netopeer2 in a not that mature fashion like you.

I understand your concerns not to do the installation at build time. But without that currently it is not that easy to get a working installation and it would be hard for users of this package to get it up and running. So my idea is to have the module installation optional in the first step. With that we can switch between having an out of the box working netopeer2 installation and the possibility to do the configuration/installation at startup like you explained above.

Thank you very much.

Signature · 1 line
Heiko
#9 lore raw

PATCH 1/2 package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation

In reply to Heiko Thiery

Heiko, All,

On 2021-02-06 11:57 +0100, Heiko Thiery spake thusly:

Quoted 11 lines

On install step the host tool syrepoctl is used to install some YANG modules. Unfortunatly syrepoctl creates some files in /dev/shm folder and does not cleanup afterwards. This files can be incompatible depending on the used sysrepo version. This causes autobuilder failures when updating the package [1].

To make sure we can remove this leftovers of sysrepoctl we specify a build specific SYSREPO_SHM_PREFIX. With this the files can deleted safely after installation is completed. This also ensures that concurrent parallel builds will not affected mutualy.

Quoted 5 lines

Fixes:
[1] http://autobuild.buildroot.net/results/6e559c4f98b7ed93d7b5af638264e907492a6532/

Series of two applied to master, thanks.

I inverted the order in which the two patches are applied, because I found it to be more logical: we need host-sysrepo, and then we can fix its usage...

As for the now-second patch, I did what I said in my previous review, plus a few litle other cleanuos and additions:

- also use the package name as discriminant
- expand commit log accordingly
- rename the variable to start with the package name
- explain why we clean up before as well

Thanks! :-)

Regards,
Yann E. MORIN.

Quoted 39 lines
---
 package/netopeer2/netopeer2.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
package/netopeer2/netopeer2.mk +14 -0
index bc02e0dc93..af30d8417a 100644
@@ -13,9 +13,23 @@ NETOPEER2_DEPENDENCIES = libnetconf2 libyang sysrepo
 
 NETOPEER2_CONF_OPTS = -DBUILD_CLI=$(if $(BR2_PACKAGE_NETOPEER2_CLI),ON,OFF)
 
+# Set a build specific SYSREPO_SHM_PREFIX to ensure we can safely delete the
+# files. This also ensures that concurrent parallel builds will not be
+# affected mutualy.
+SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))
+NETOPEER2_MAKE_ENV = SYSREPO_SHM_PREFIX=$(SYSREPO_SHM_PREFIX)
+
 define NETOPEER2_INSTALL_INIT_SYSV
 	$(INSTALL) -m 755 -D package/netopeer2/S52netopeer2 \
 		$(TARGET_DIR)/etc/init.d/S52netopeer2
 endef
 
+# The host sysrepo used to install the netopeer2 modules will leave
+# its shared memory files lingering about. Clean up in it's stead...
+define NETOPEER2_CLEANUP
+   rm -f /dev/shm/$(SYSREPO_SHM_PREFIX)*
+endef
+NETOPEER2_PRE_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP
+NETOPEER2_POST_INSTALL_TARGET_HOOKS += NETOPEER2_CLEANUP
+
 $(eval $(cmake-package))
Signature · 6 lines
2.20.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'