Fix fail2ban
cover Fix fail2ban
This series fixes fail2ban not installing configuration files in /etc anymore. While fixing the problem, I discovered a bug in the way we invoke the pyinstaller.py in pep517 compatibility.
Basically, pyinstaller.py will install data_files defined in setup.py and tarballed in the wheel inside the directory pointed by the --data parameter, which is nowadays pointing to /usr of TARGET/STAGING directory.
I consider this wrong, because the data folder shall point to the TARGET_DIR/STAGING_DIR to be correct and we shall trust the destination path expressed in the setup.py.
In the case of fail2ban, configuration files end up being installed in /usr/etc/fail2ban instead of /etc/fail2ban.
v1 -> v2:
Fix signed off with correct email.
v2 -> v3:
Fix commit author too
Angelo Compagnucci (2):
package/fail2ban: fix data_files installation path package/pkg-python: fix pep517 data files installation
...001-setup.py-fix-data-relative-paths.patch | 73 +++++++++++++++++++ package/pkg-python.mk | 4 +- 2 files changed, 75 insertions(+), 2 deletions(-) create mode 100644 package/fail2ban/0001-setup.py-fix-data-relative-paths.patch
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 1/2 package/fail2ban: fix data_files installation path
When packaging data_files in a wheel, those files shall have the destination path expressed as relative to the wheel installation.
The reason for that is the wheel installation shall be contained inside a specific folder and/or in site-packages.
When building the wheel file by the bdist_wheel command, python will package files with absolute pats as python files ending up installing them in site-packages.
Fixes:
http://autobuild.buildroot.net/results/36ac5278d19195a21c3d02d087965e08f49228ef
--- ...001-setup.py-fix-data-relative-paths.patch | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 package/fail2ban/0001-setup.py-fix-data-relative-paths.patch
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch new file +73 -0
@@ -0,0 +1,73 @@ +From 7b7fa3c02d4575c09362336bc310835c37859f1a Mon Sep 17 00:00:00 2001 +From: Angelo Compagnucci <[email protected]> +Date: Fri, 19 Apr 2024 22:47:20 +0200 +Subject: [PATCH] setup.py: fix data relative paths +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +As per distutils documentation: + +The directory should be a relative path. It is interpreted relative to the +installation prefix (Python’s sys.prefix for system installations; +site.USER_BASE for user installations). Distutils allows directory to be an +absolute installation path, but this is discouraged since it is incompatible +with the wheel packaging format [1]. + +When packaging to a bdist_wheel indeed, the files in the "data_files" dictionary +are not picked at all as data files if they have an absolute path, +but they are instead added to the wheel as python files. +This has the effect of installing them as part of python files in the +site-packages folder when the wheel is installed. + +This patch cannot be sent upstream because it breaks the legacy way of +installing the package with *python setup.py install* and it is needed only to +comply with the pep517 buildroot compatibility layer. + +[1] https://docs.python.org/3.10/distutils/setupscript.html#installing-additional-files +Signed-off-by: Angelo Compagnucci <[email protected]> +--- + setup.py | 14 +++++++------- + 1 file changed, 7 insertions(+), 7 deletions(-) + +diff --git a/setup.py b/setup.py +index 91f71cf2..99ba2e32 100755 +--- a/setup.py ++++ b/setup.py +@@ -238,26 +238,26 @@ setup( + for f in w[2]] + } if with_tests else {}, + data_files = [ +- ('/etc/fail2ban', ++ ('etc/fail2ban', + glob("config/*.conf") + ), +- ('/etc/fail2ban/filter.d', ++ ('etc/fail2ban/filter.d', + glob("config/filter.d/*.conf") + ), +- ('/etc/fail2ban/filter.d/ignorecommands', ++ ('etc/fail2ban/filter.d/ignorecommands', + [p for p in glob("config/filter.d/ignorecommands/*") if isfile(p)] + ), +- ('/etc/fail2ban/action.d', ++ ('etc/fail2ban/action.d', + glob("config/action.d/*.conf") + + glob("config/action.d/*.py") + ), +- ('/etc/fail2ban/fail2ban.d', ++ ('etc/fail2ban/fail2ban.d', + '' + ), +- ('/etc/fail2ban/jail.d', ++ ('etc/fail2ban/jail.d', + '' + ), +- ('/var/lib/fail2ban', ++ ('var/lib/fail2ban', + '' + ), + ] + data_files_extra, +-- +2.34.1 +
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 1/2] package/fail2ban: fix data_files installation path
Am 19.04.24 um 23:32 schrieb Angelo Compagnucci:
Quoted 8 lines
When packaging
data_filesin a wheel, those files shall have the destination path expressed as relative to the wheel installation.
The reason for that is the wheel installation shall be contained inside a specific folder and/or in site-packages.
When building the wheel file by thebdist_wheelcommand, python will package files with absolute pats as python files ending up installing them in site-packages.
Quoted 4 lines
Fixes:
http://autobuild.buildroot.net/results/36ac5278d19195a21c3d02d087965e08f49228efSigned-off-by Angelo Compagnucci <[email protected]>
Testing included: Build for aarch64, configuring fail2ban for sshd, testing that a host indeed gets banned after repeated failed login attempts (password auth), with syslog-ng (the default fail2ban patterns don't work with Busybox syslogd) and nftables.
Thanks for fixing the build, Angelo!
Best regards,
Fiona
Quoted 74 lines
--- ...001-setup.py-fix-data-relative-paths.patch | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+)create mode 100644
package/fail2ban/0001-setup.py-fix-data-relative-paths.patchpackage/fail2ban/0001-setup.py-fix-data-relative-paths.patch new file +63
-0@@ -0,0 +1,73 @@ +From 7b7fa3c02d4575c09362336bc310835c37859f1a Mon Sep 17 00:00:00 2001 +From: Angelo Compagnucci <[email protected]> +Date: Fri, 19 Apr 2024 22:47:20 +0200 +Subject: [PATCH] setup.py: fix data relative paths +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +As per distutils documentation: + +The directory should be a relative path. It is interpreted relative to the +installation prefix (Python’s sys.prefix for system installations; +site.USER_BASE for user installations). Distutils allows directory to be an +absolute installation path, but this is discouraged since it is incompatible +with the wheel packaging format [1]. + +When packaging to a bdist_wheel indeed, the files in the "data_files" dictionary +are not picked at all as data files if they have an absolute path, +but they are instead added to the wheel as python files. +This has the effect of installing them as part of python files in the +site-packages folder when the wheel is installed. + +This patch cannot be sent upstream because it breaks the legacy way of +installing the package with *python setup.py install* and it is needed only to +comply with the pep517 buildroot compatibility layer. + +[1] https://docs.python.org/3.10/distutils/setupscript.html#installing-additional-files +Signed-off-by: Angelo Compagnucci <[email protected]> +--- + setup.py | 14 +++++++------- + 1 file changed, 7 insertions(+), 7 deletions(-) + +diff --git a/setup.py b/setup.py +index 91f71cf2..99ba2e32 100755 +--- a/setup.py ++++ b/setup.py +@@ -238,26 +238,26 @@ setup( + for f in w[2]] + } if with_tests else {}, + data_files = [ +- ('/etc/fail2ban', ++ ('etc/fail2ban', + glob("config/*.conf") + ), +- ('/etc/fail2ban/filter.d', ++ ('etc/fail2ban/filter.d', + glob("config/filter.d/*.conf") + ), +- ('/etc/fail2ban/filter.d/ignorecommands', ++ ('etc/fail2ban/filter.d/ignorecommands', + [p for p in glob("config/filter.d/ignorecommands/*") if isfile(p)] + ), +- ('/etc/fail2ban/action.d', ++ ('etc/fail2ban/action.d', + glob("config/action.d/*.conf") + + glob("config/action.d/*.py") + ), +- ('/etc/fail2ban/fail2ban.d', ++ ('etc/fail2ban/fail2ban.d', + '' + ), +- ('/etc/fail2ban/jail.d', ++ ('etc/fail2ban/jail.d',
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch line 73
Quoted 10 lines
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch new file +10
-0+ '' + ), +- ('/var/lib/fail2ban', ++ ('var/lib/fail2ban', + '' + ), + ] + data_files_extra, +-- +2.34.1 +
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 1/2] package/fail2ban: fix data_files installation path
In reply to Angelo CompagnucciHello Angelo,
Quoted 24 lines · On Fri, 19 Apr 2024 23:32:15 +0200 Angelo Compagnucci…
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch new file +18
-0@@ -0,0 +1,73 @@ +From 7b7fa3c02d4575c09362336bc310835c37859f1a Mon Sep 17 00:00:00 2001 +From: Angelo Compagnucci <[email protected]> +Date: Fri, 19 Apr 2024 22:47:20 +0200 +Subject: [PATCH] setup.py: fix data relative paths +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +As per distutils documentation: + +The directory should be a relative path. It is interpreted relative to the +installation prefix (Python’s sys.prefix for system installations; +site.USER_BASE for user installations). Distutils allows directory to be an +absolute installation path, but this is discouraged since it is incompatible +with the wheel packaging format [1]. + +When packaging to a bdist_wheel indeed, the files in the "data_files" dictionary +are not picked at all as data files if they have an absolute path,
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch line 28
Quoted 10 lines
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch new file +10
-0+but they are instead added to the wheel as python files. +This has the effect of installing them as part of python files in the +site-packages folder when the wheel is installed. + +This patch cannot be sent upstream because it breaks the legacy way of +installing the package with *python setup.py install* and it is needed only to +comply with the pep517 buildroot compatibility layer. + +[1] https://docs.python.org/3.10/distutils/setupscript.html#installing-additional-files +Signed-off-by: Angelo Compagnucci <[email protected]>
Thanks for the patch!
However, this patch has no Upstream: tag, causing a check-package warning when being applied:
WARNING: package/fail2ban/0001-setup.py-fix-data-relative-paths.patch:0: missing Upstream in the header (https://nightly.buildroot.org/#_additional_patch_documentation)
What is the upstream status of this patch?
Thanks!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 1/2] package/fail2ban: fix data_files installation path
Am 09.07.24 um 22:28 schrieb Thomas Petazzoni via buildroot:
Quoted 43 lines
Hello Angelo,
Quoted 24 lines · On Fri, 19 Apr 2024 23:32:15 +0200 Angelo Compagnucci…
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch new file +18
-0@@ -0,0 +1,73 @@ +From 7b7fa3c02d4575c09362336bc310835c37859f1a Mon Sep 17 00:00:00 2001 +From: Angelo Compagnucci <[email protected]> +Date: Fri, 19 Apr 2024 22:47:20 +0200 +Subject: [PATCH] setup.py: fix data relative paths +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +As per distutils documentation: + +The directory should be a relative path. It is interpreted relative to the +installation prefix (Python’s sys.prefix for system installations; +site.USER_BASE for user installations). Distutils allows directory to be an +absolute installation path, but this is discouraged since it is incompatible +with the wheel packaging format [1]. + +When packaging to a bdist_wheel indeed, the files in the "data_files" dictionary +are not picked at all as data files if they have an absolute path,package/fail2ban/0001-setup.py-fix-data-relative-paths.patch line 28
Quoted 10 lines
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch new file +10
-0+but they are instead added to the wheel as python files. +This has the effect of installing them as part of python files in the +site-packages folder when the wheel is installed. + +This patch cannot be sent upstream because it breaks the legacy way of +installing the package with *python setup.py install* and it is needed only to +comply with the pep517 buildroot compatibility layer. + +[1] https://docs.python.org/3.10/distutils/setupscript.html#installing-additional-files +Signed-off-by: Angelo Compagnucci <[email protected]>Thanks for the patch!
Quoted 6 lines
However, this patch has no Upstream: tag, causing a check-package warning when being applied:
WARNING:
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch:0: missing Upstream in the header (https://nightly.buildroot.org/#_additional_patch_documentation)What is the upstream status of this patch?
I believe the last paragraph of the patch description answers the question, maybe you could add a shortened version as the Upstream tag? E.g.:
Upstream: N/A (breaks upstream install, only for Buildroot compatibility)
Best regards,
Fiona
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/2 package/pkg-python: fix pep517 data files installation
In reply to Angelo CompagnucciWhen installing a wheel in pep517 compatibility mode, the pyinstaller.py is invoked with the --data parameter which point to the directory in which data files should be installed.
Actually the --data parameter points to the /usr subdirectory which is indeed wrong cause it shall point to the root directory where the wheel will be installed.
This fixes the problem of having configuration files installed in /usr/etc instead of /etc.
Fixes:
http://autobuild.buildroot.net/results/36ac5278d19195a21c3d02d087965e08f49228ef
--- package/pkg-python.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
package/pkg-python.mk +2 -2
@@ -70,7 +70,7 @@ PKG_PYTHON_PEP517_INSTALL_TARGET_CMD = \ --purelib=$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ --headers=$(TARGET_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR) \ --scripts=$(TARGET_DIR)/usr/bin \ - --data=$(TARGET_DIR)/usr + --data=$(TARGET_DIR) PKG_PYTHON_PEP517_INSTALL_STAGING_CMD = \ $(TOPDIR)/support/scripts/pyinstaller.py \
@@ -79,7 +79,7 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_CMD = \ --purelib=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ --headers=$(STAGING_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR) \ --scripts=$(STAGING_DIR)/usr/bin \ - --data=$(STAGING_DIR)/usr + --data=$(STAGING_DIR) PKG_PYTHON_PEP517_DEPENDENCIES = \ host-python-pypa-build \
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/pkg-python: fix pep517 data files installation
Am 19.04.24 um 23:32 schrieb Angelo Compagnucci:
Quoted 9 lines
When installing a wheel in pep517 compatibility mode, the
pyinstaller.pyis invoked with the --data parameter which point to the directory in which data files should be installed.
Actually the --data parameter points to the /usr subdirectory which is indeed wrong cause it shall point to the root directory where the wheel will be installed.
This fixes the problem of having configuration files installed in/usr/etcinstead of /etc.
Quoted 4 lines
Fixes:
http://autobuild.buildroot.net/results/36ac5278d19195a21c3d02d087965e08f49228efSigned-off-by Angelo Compagnucci <[email protected]>
Testing included: Build for aarch64, configuring fail2ban for sshd, testing that a host indeed gets banned after repeated failed login attempts (password auth), with syslog-ng (the default fail2ban patterns don't work with Busybox syslogd) and nftables.
Quoted 18 lines
--- package/pkg-python.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)package/pkg-python.mk +1
-1@@ -70,7 +70,7 @@ PKG_PYTHON_PEP517_INSTALL_TARGET_CMD = \ --purelib=$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ --headers=$(TARGET_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR) \ --scripts=$(TARGET_DIR)/usr/bin \ - --data=$(TARGET_DIR)/usr + --data=$(TARGET_DIR) PKG_PYTHON_PEP517_INSTALL_STAGING_CMD = \ $(TOPDIR)/support/scripts/pyinstaller.py \
package/pkg-python.mk PKG_PYTHON_PEP517_INSTALL_STAGING_CMD = \ · line 85
Quoted 8 lines
package/pkg-python.mk +1
-1@@ -79,7 +79,7 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_CMD = \ --purelib=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ --headers=$(STAGING_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR) \ --scripts=$(STAGING_DIR)/usr/bin \ - --data=$(STAGING_DIR)/usr + --data=$(STAGING_DIR) PKG_PYTHON_PEP517_DEPENDENCIES = \ host-python-pypa-build \
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 20 lines · Angelo Compagnucci
This series fixes fail2ban not installing configuration files in /etc anymore. While fixing the problem, I discovered a bug in the way we invoke the
pyinstaller.pyin pep517 compatibility.
Basically,pyinstaller.pywill installdata_filesdefined insetup.pyand tarballed in the wheel inside the directory pointed by the --data parameter, which is nowadays pointing to /usr of TARGET/STAGING directory.I consider this wrong, because the data folder shall point to the TARGET_DIR/STAGING_DIR to be correct and we shall trust the destination path expressed in the
setup.py.In the case of fail2ban, configuration files end up being installed in
/usr/etc/fail2baninstead of/etc/fail2ban.v1 -> v2:
Fix signed off with correct email.
v2 -> v3:
Fix commit author too
Quoted 3 lines
Angelo Compagnucci (2):
package/fail2ban: fixdata_filesinstallation path package/pkg-python: fix pep517 data files installation
Applied both to master, thanks.
I added the Upstream: tag as suggested by Fiona. Still, I think it would be
good to report the issue upstream, so they can take it into account when they eventually get around to migrating to a pep517 build system.
I also changed the order of the two commits, and removed the Fixes entry from
the one in pkg-python.
Do you know if this issue also occurs on 2024.02.x and 2024.05.x?
Regards,
Arnout
Quoted 6 lines
...001-setup.py-fix-data-relative-paths.patch | 73 +++++++++++++++++++ package/pkg-python.mk | 4 +- 2 files changed, 75 insertions(+), 2 deletions(-)create mode 100644
package/fail2ban/0001-setup.py-fix-data-relative-paths.patch
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Il giorno ven 12 lug 2024 alle ore 15:42 Arnout Vandecappelle < [email protected]> ha scritto:
Quoted 38 lines · Angelo Compagnucci
Quoted 1 line
This series fixes fail2ban not installing configuration files in /etc
anymore.
Quoted 4 lines
While fixing the problem, I discovered a bug in the way we invoke the
pyinstaller.pyin pep517 compatibility.
Basically,pyinstaller.pywill installdata_filesdefined insetup.pyand tarballed in the wheel inside the directory pointed by the --dataparameter,
Quoted 4 lines
which is nowadays pointing to /usr of TARGET/STAGING directory.
I consider this wrong, because the data folder shall point to the TARGET_DIR/STAGING_DIR to be correct and we shall trust the destination
path
Quoted 11 lines
expressed in the
setup.py.In the case of fail2ban, configuration files end up being installed in
/usr/etc/fail2baninstead of/etc/fail2ban.v1 -> v2:
Fix signed off with correct email.
v2 -> v3:
Fix commit author tooQuoted 3 lines
Angelo Compagnucci (2):
package/fail2ban: fixdata_filesinstallation path package/pkg-python: fix pep517 data files installationApplied both to master, thanks.
I added the Upstream: tag as suggested by Fiona. Still, I think it would
be
good to report the issue upstream, so they can take it into account when they
eventually get around to migrating to a pep517 build system.
Quoted 6 lines
I also changed the order of the two commits, and removed the Fixes entry
from
the one in pkg-python.Do you know if this issue also occurs on 2024.02.x and 2024.05.x?
I sent the patch on April 19 to silence the numerous package failings I was already receiving. Not sure for the 2024.05, but surely the 2024.05 is affected.
Quoted 12 lines
Regards,
ArnoutQuoted 5 lines
...001-setup.py-fix-data-relative-paths.patch | 73 +++++++++++++++++++ package/pkg-python.mk | 4 +- 2 files changed, 75 insertions(+), 2 deletions(-)create mode 100644
package/fail2ban/0001-setup.py-fix-data-relative-paths.patchQuoted 1 line
Signature · 1 line
Profile: http://it.linkedin.com/in/compagnucciangelo
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Angelo" == Angelo Compagnucci <[email protected]> writes:
Hi,
Quoted 6 lines
Quoted 2 lines
Do you know if this issue also occurs on 2024.02.x and 2024.05.x?
I sent the patch on April 19 to silence the numerous package failings I was already receiving. Not sure for the 2024.05, but surely the 2024.05 is affected.
I was unable to reproduce it on 2024.02.x, but it indeed also fails on 2024.05.x. I believe it was introduced by commit 8937db8dd595e (package/pkg-python.mk: migrate setuptools to pep517).
So committed both to 2024.05.x, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot