Skip to contentSkip to conversation

package/filebeat: new package

PATCH In review [email protected] 2 participants 6 messages Nov 9, 2023 – Feb 4, 2025

lore
Share & tools
lore
#0 lore raw

cover package/filebeat: new package

From: "Yann E. MORIN" <[email protected]>

Hello All!

This two-patch series introduces filebeat, an ElasticSearch-related tool to grab local log files and send them to a remote ElasticSearch server.

It is a bit convoluted (see commit log), and requires a little extension in our golang-package infrastructure, which we introduce as the first patch in the series.

$ echo BR2_PACKAGE_FILEBEAT=y >filebeat.cfg
$ ./utils/test-pkg -c filebeat.cfg -p filebeat -d $(pwd)/test-pkg -a
                             arm-aarch64 [ 1/45]: OK
                   bootlin-aarch64-glibc [ 2/45]: OK
               bootlin-arcle-hs38-uclibc [ 3/45]: SKIPPED
                    bootlin-armv5-uclibc [ 4/45]: OK
                     bootlin-armv7-glibc [ 5/45]: OK
                   bootlin-armv7m-uclibc [ 6/45]: SKIPPED
                      bootlin-armv7-musl [ 7/45]: OK
                bootlin-m68k-5208-uclibc [ 8/45]: SKIPPED
               bootlin-m68k-68040-uclibc [ 9/45]: SKIPPED
             bootlin-microblazeel-uclibc [10/45]: SKIPPED
                bootlin-mipsel32r6-glibc [11/45]: SKIPPED
                   bootlin-mipsel-uclibc [12/45]: SKIPPED
                     bootlin-nios2-glibc [13/45]: SKIPPED
                 bootlin-openrisc-uclibc [14/45]: SKIPPED
        bootlin-powerpc64le-power8-glibc [15/45]: OK
           bootlin-powerpc-e500mc-uclibc [16/45]: SKIPPED
                   bootlin-riscv32-glibc [17/45]: SKIPPED
                   bootlin-riscv64-glibc [18/45]: SKIPPED
                    bootlin-riscv64-musl [19/45]: SKIPPED
                 bootlin-s390x-z13-glibc [20/45]: OK
                      bootlin-sh4-uclibc [21/45]: SKIPPED
                   bootlin-sparc64-glibc [22/45]: SKIPPED
                    bootlin-sparc-uclibc [23/45]: SKIPPED
                    bootlin-x86-64-glibc [24/45]: OK
                     bootlin-x86-64-musl [25/45]: OK
                   bootlin-x86-64-uclibc [26/45]: OK
                   bootlin-xtensa-uclibc [27/45]: SKIPPED
                            br-arm-basic [28/45]: OK
                    br-arm-full-nothread [29/45]: OK
                      br-arm-full-static [30/45]: SKIPPED
                   br-i386-pentium4-full [31/45]: OK
                br-i386-pentium-mmx-musl [32/45]: OK
                      br-mips64-n64-full [33/45]: OK
                 br-mips64r6-el-hf-glibc [34/45]: SKIPPED
               br-powerpc-603e-basic-cpp [35/45]: SKIPPED
               br-powerpc64-power7-glibc [36/45]: SKIPPED
                       linaro-aarch64-be [37/45]: SKIPPED
                          linaro-aarch64 [38/45]: OK
                              linaro-arm [39/45]: OK
                     sourcery-arm-armv4t [40/45]: SKIPPED
                            sourcery-arm [41/45]: FAILED
                     sourcery-arm-thumb2 [42/45]: FAILED
                         sourcery-mips64 [43/45]: FAILED
                           sourcery-mips [44/45]: SKIPPED
                          sourcery-nios2 [45/45]: SKIPPED
45 builds, 23 skipped, 5 build failed, 0 legal-info failed, 0 show-info failed

Analysis of each build failure:

- sourcery-arm, sourcery-arm-thumb2:
=> the failure with the sourcery-arm toolchain is expected: it is very old, and uses binutils that did not not have a ld.gold, so no ld.bfd either, just a plain ld, hence our workaround against the go workaround does not work, as expected.

- sourcery-mips64:
=> host-go does not build, so not a filebeat issue

- bootlin-riscv64-glibc, bootlin-riscv64-musl:
=> filebeat has no implementation for its internal atomic ints on riscv64

- other skips due to go arch dependencies: no go, no beat.

Regards,
Yann E. MORIN.

The following changes since commit a01490397e4cc11155b3c396ded382eefeef4f12

package/opencv4: security bump to version 4.8.1 (2023-11-08 23:28:53 +0100)

are available as patches in this mail series,

for you to apply patches up to 46f20e86fa6c17ada33ede672ffe4229d8bf26a3

package/filebeat: new package (2023-11-09 17:39:20 +0100)

----------------------------------------------------------------
Yann E. MORIN (2):
package/pkg-golang: allow packages to provide extldflags package/filebeat: new package

 DEVELOPERS                              |  1 +
 docs/manual/adding-packages-golang.adoc |  6 ++--
 package/Config.in                       |  1 +
 package/filebeat/Config.in              | 14 +++++++++
 package/filebeat/filebeat.default       |  1 +
 package/filebeat/filebeat.hash          |  5 ++++
 package/filebeat/filebeat.mk            | 52 +++++++++++++++++++++++++++++++++
 package/filebeat/filebeat.service       | 13 +++++++++
 package/filebeat/filebeat.yml           |  5 ++++
 package/pkg-golang.mk                   |  6 +++-
 10 files changed, 101 insertions(+), 3 deletions(-)
 create mode 100644 package/filebeat/Config.in
 create mode 100644 package/filebeat/filebeat.default
 create mode 100644 package/filebeat/filebeat.hash
 create mode 100644 package/filebeat/filebeat.mk
 create mode 100644 package/filebeat/filebeat.service
 create mode 100644 package/filebeat/filebeat.yml

--

                                        ____________
.-----------------.--------------------:       _    :------------------.
|  Yann E. MORIN  | Real-Time Embedded |    __/ )   | /"\ ASCII RIBBON |
|                 | Software  Designer |  _/ - /'   | \ / CAMPAIGN     |
| +33 638.411.245 '--------------------: (_    `--, |  X  AGAINST      |
| yann.morin (at) orange.com           |_="    ,--' | / \ HTML MAIL    |

'--------------------------------------:______/_____:------------------'

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#1 lore raw

PATCH 1/2 package/pkg-golang: allow packages to provide extldflags

From: "Yann E. MORIN" <[email protected]>

For some packages, it may be necessary to pass extra extldflags. For example, some packages will need to be able to specify what ld to use, and so will need to pass -fuse-ld={bfd,gold} as extldflags.

Introduce FOO_EXTLDFLAGS, that packages can set to pass such flags.

---
 docs/manual/adding-packages-golang.adoc | 6 ++++--
 package/pkg-golang.mk                   | 6 +++++-
 2 files changed, 9 insertions(+), 3 deletions(-)
docs/manual/adding-packages-golang.adoc +4 -2
index c952cde577..c238050096 100644
@@ -84,8 +84,10 @@ therefore only use a few of them, or none.
   infrastructure will automatically generate a minimal +go.mod+ file
   in the package source tree if it doesn't exist.
 
-* +FOO_LDFLAGS+ and +FOO_TAGS+ can be used to pass respectively the
-  +LDFLAGS+ or the +TAGS+ to the +go+ build command.
+* +FOO_LDFLAGS+, +FOO_EXTLDFLAGS+, and +FOO_TAGS+ can be used to pass
+  respectively the go +LDFLAGS+ (via the `-ldflags` command line flag),
+  the external linker flags +EXTLDFLAGS+ (via the `-extldflags`
+  command line flag), or the +TAGS+ to the +go+ build command.
 
 * +FOO_BUILD_TARGETS+ can be used to pass the list of targets that
   should be built. If +FOO_BUILD_TARGETS+ is not specified, it
package/pkg-golang.mk +5 -1
index 8e27602d41..206688fa7b 100644
@@ -114,10 +114,14 @@ ifndef $(2)_BUILD_CMDS
 ifeq ($(4),target)
 
 ifeq ($(BR2_STATIC_LIBS),y)
-$(2)_LDFLAGS += -extldflags '-static'
+$(2)_EXTLDFLAGS += -static
 $(2)_TAGS += osusergo netgo
 endif
 
+ifneq ($$($(2)_EXTLDFLAGS),)
+$(2)_LDFLAGS += -extldflags '$$($(2)_EXTLDFLAGS)'
+endif
+
 # Build package for target
 define $(2)_BUILD_CMDS
 	$$(foreach d,$$($(2)_BUILD_TARGETS),\
Signature · 18 lines
2.34.1

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Tested-by Fiona Klute (WIWA) on v1 1/2
#2 lore raw

Re: [Buildroot] [PATCH 1/2] package/pkg-golang: allow packages to provide extldflags

Am 09.11.23 um 17:39 schrieb [email protected]:

Quoted 10 lines

From: "Yann E. MORIN" <[email protected]>

For some packages, it may be necessary to pass extra extldflags. For example, some packages will need to be able to specify what ld to use, and so will need to pass -fuse-ld={bfd,gold} as extldflags.

Introduce FOO_EXTLDFLAGS, that packages can set to pass such flags.

Thanks,
Fiona

Quoted 32 lines
---
  docs/manual/adding-packages-golang.adoc | 6 ++++--
  package/pkg-golang.mk                   | 6 +++++-
  2 files changed, 9 insertions(+), 3 deletions(-)
docs/manual/adding-packages-golang.adoc +4 -2
index c952cde577..c238050096 100644
@@ -84,8 +84,10 @@ therefore only use a few of them, or none.
    infrastructure will automatically generate a minimal +go.mod+ file
    in the package source tree if it doesn't exist.
 
-* +FOO_LDFLAGS+ and +FOO_TAGS+ can be used to pass respectively the
-  +LDFLAGS+ or the +TAGS+ to the +go+ build command.
+* +FOO_LDFLAGS+, +FOO_EXTLDFLAGS+, and +FOO_TAGS+ can be used to pass
+  respectively the go +LDFLAGS+ (via the `-ldflags` command line flag),
+  the external linker flags +EXTLDFLAGS+ (via the `-extldflags`
+  command line flag), or the +TAGS+ to the +go+ build command.
 
  * +FOO_BUILD_TARGETS+ can be used to pass the list of targets that
    should be built. If +FOO_BUILD_TARGETS+ is not specified, it
package/pkg-golang.mk +1 -1
index 8e27602d41..206688fa7b 100644
@@ -114,10 +114,14 @@ ifndef $(2)_BUILD_CMDS
  ifeq ($(4),target)
 
  ifeq ($(BR2_STATIC_LIBS),y)
-$(2)_LDFLAGS += -extldflags '-static'
+$(2)_EXTLDFLAGS += -static

package/pkg-golang.mk ifndef $(2)_BUILD_CMDS · line 127

Quoted 10 lines
package/pkg-golang.mk +4 -0
  $(2)_TAGS += osusergo netgo
  endif
 
+ifneq ($$($(2)_EXTLDFLAGS),)
+$(2)_LDFLAGS += -extldflags '$$($(2)_EXTLDFLAGS)'
+endif
+
  # Build package for target
  define $(2)_BUILD_CMDS
  	$$(foreach d,$$($(2)_BUILD_TARGETS),\

--
Dipl.-Ing. Fiona Klute
Mollwitzer Str. 2
44141 Dortmund
Germany

USt.-ID/VAT number: DE363488944
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#3 lore raw

PATCH 2/2 package/filebeat: new package

In reply to [email protected]

From: "Yann E. MORIN" <[email protected]>

filebeat is one of the many Elastic 'beats' that can send data to a remote ElasticSearch (ES) engine or to a remote logstash.

This is a go package, but the ES developers are using layers upon layers of abstractions to try and fail to provide an easier build mechanism.

First, they use a convoluted set of Makefiles to wrap a lot of arcane commands. There is no obvious way to follow what's going on, especially as...

... second they use mage [0] as a first layer to abstract 'go build', but mage really does not play nicely in cross-compilation; it is a PITA to use, and it very easily gets confused. Building natively is also very fragile and heavily dependent on the user's environment (env, path, tools versions...), which makes it fails quite easily and spectacularly...

But in the end, a simple "go build" works like a charm to build the beats' executable. This means we can just use Buildroot's golang-package infrastructure. Wee!

On AArch64, the link may fail with errors like:

[...]
github.com/elastic/beats/v7/filebeat
# github.com/elastic/beats/v7/filebeat
[...]/output/per-package/filebeat/host/lib/go/pkg/tool/linux_amd64/link: running [...]/output/per-package/filebeat/host/bin/aarch64-linux-gcc failed: exit status 1
collect2: fatal error: cannot find ‘ld’
compilation terminated.

The error happens because go, on AArch64, will forcibly want to use ld.gold, as a workaround for a alleged binutils breakage, as reported upstreams [1] [2] [3] [4].

This is supposedly fixed in binutils 2.36, but go still wants to use ld.gold. As not all toolchains have gold, force use of ld.bfd, which in practice has been shown to work correctly with recent-ish binutils
versions: the link succeeds, and the program runs. This may still break on older toolchains, especially those with older binutils that did not have ld.gold (so they would not have ld.bfd either, just ld), but better have breakage on older toolchains, for which there's not much we can do, than on newer ones.

(the same issue also occurred for ARM, but the ld.gold requirement was dropped in cd77738198ff, first included in go 1.21, which is now what we use in Buildroot)

On riscv64, the build fails because of missing atomic ints:
github.com/elastic/beats/v7/libbeat/statestore github.com/elastic/beats/v7/filebeat/inputsource/common/streaming # github.com/elastic/beats/v7/libbeat/statestore libbeat/statestore/store.go:28:18: undefined: atomic.Int libbeat/statestore/store.go:48:20: undefined: atomic.MakeInt github.com/elastic/beats/v7/libbeat/processors/ratelimit github.com/elastic/beats/v7/filebeat/input/kafka
# github.com/elastic/beats/v7/filebeat/inputsource/common/streaming filebeat/inputsource/common/streaming/listener.go:48:25: undefined: atomic.Int # github.com/elastic/beats/v7/libbeat/processors/ratelimit libbeat/processors/ratelimit/token_bucket.go:53:20: undefined: atomic.Uint libbeat/processors/ratelimit/token_bucket.go:96:21: undefined: atomic.Uint libbeat/processors/ratelimit/token_bucket.go:193:34: undefined: atomic.MakeUint # github.com/elastic/beats/v7/filebeat/input/kafka filebeat/input/kafka/input.go:395:23: undefined: atomic.MakeInt

That's because beats reimplement/wrap atomic ints, but the file implementing those is not whitelisted for riscv64 [1] (yes, 7.0 because this is what kafka's input module imports even in 8.8.2) and it has still not changed in 8.8.2 anyway [2] (thanks to Vincent Fazio for pointing those out).

Running in an sysv-init environment is untested, and left as an exercise for interested parties.

Finally, we only package version 8.8.2, because any later version can't be vendored [3]; it's been fixed [4] but is not part of any release yet.

[0] https://magefile.org/
[1] https://github.com/golang/go/commit/3c8d6af8e02bbf230c2bef9f181d8ea393068299
[2] https://github.com/golang/go/issues/22040
[3] https://go-review.googlesource.com/c/go/+/22141
[4] https://sourceware.org/bugzilla/show_bug.cgi?id=19962
[5] https://github.com/elastic/beats/blame/7.0/libbeat/common/atomic/atomic64.go#L18
[6] https://github.com/elastic/beats/blame/v8.2.2/libbeat/common/atomic/atomic64.go#L18
[7] https://github.com/elastic/beats/issues/36949
[8] https://github.com/elastic/beats/pull/36953

---
Changes v1 -> v2:
- fix Vincent's name, with my humble apologies
- don't force ld.bfd for ARM (no longer needed)
- don't install "data files" as they are useless (Nathaniel)
- add entry to DEVELOPERS

---
 DEVELOPERS                        |  1 +
 package/Config.in                 |  1 +
 package/filebeat/Config.in        | 14 +++++++++
 package/filebeat/filebeat.default |  1 +
 package/filebeat/filebeat.hash    |  5 +++
 package/filebeat/filebeat.mk      | 52 +++++++++++++++++++++++++++++++
 package/filebeat/filebeat.service | 13 ++++++++
 package/filebeat/filebeat.yml     |  5 +++
 8 files changed, 92 insertions(+)
 create mode 100644 package/filebeat/Config.in
 create mode 100644 package/filebeat/filebeat.default
 create mode 100644 package/filebeat/filebeat.hash
 create mode 100644 package/filebeat/filebeat.mk
 create mode 100644 package/filebeat/filebeat.service
 create mode 100644 package/filebeat/filebeat.yml
DEVELOPERS +1 -0
index 12257fae8c..94d0ef7cb5 100644
@@ -3191,6 +3191,7 @@ F:	support/download/
 
 N:	Yann E. MORIN <[email protected]>
 F:	.editorconfig
+F:	package/filebeat/
 F:	package/gpsd/
 
 N:	Yegor Yefremov <[email protected]>
package/Config.in +1 -0
index ce46d30fed..bdcc7e9d60 100644
@@ -2356,6 +2356,7 @@ endif
 	source "package/fail2ban/Config.in"
 	source "package/fastd/Config.in"
 	source "package/fcgiwrap/Config.in"
+	source "package/filebeat/Config.in"
 	source "package/firewalld/Config.in"
 	source "package/flannel/Config.in"
 	source "package/fmc/Config.in"
package/filebeat/Config.in new file +14 -0
new file mode 100644
index 0000000000..78adae11cd
@@ -0,0 +1,14 @@
+config BR2_PACKAGE_FILEBEAT
+	bool "filebeat"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	depends on !BR2_RISCV_64
+	help
+	  Lightweight data shippers
+
+	  Beats is a free and open platform for single-purpose data
+	  shippers. They send data from hundreds or thousands of
+	  machines and systems to Logstash or Elasticsearch.
+
+	  This package only installs filebeat.
+
+	  https://www.elastic.co/products/beats
package/filebeat/filebeat.default new file +1 -0
new file mode 100644
index 0000000000..ab51c81018
@@ -0,0 +1 @@
+FILEBEAT_OPTS=-c /etc/filebeat/filebeat.yml
package/filebeat/filebeat.hash new file +5 -0
new file mode 100644
index 0000000000..11a86f81f2
@@ -0,0 +1,5 @@
+# Locally computed
+sha256  713d109760bbdfcdb30651861bd3899945114325ed65cb2bd744d7ff02c399a9  filebeat-8.8.2.tar.gz
+sha256  23e50a1dead216922519026a878d3c75cc2860eb4ca6de186ac64100f85623f3  filebeat-8.8.2-linux-x86_64.tar.gz
+sha256  7f0eb504dd0b8b090b52cff384296249be3168374f271b142ce03130c987c8e5  LICENSE.txt
+sha256  928b3ded5ea8db33fadb8e5bea8bb10e1f79d2a904e9c97745f54b57e6a1588b  NOTICE.txt
package/filebeat/filebeat.mk new file +52 -0

52 lines not on this page · raw message

package/filebeat/filebeat.service new file +13 -0
new file mode 100644
index 0000000000..08234f50b6
@@ -0,0 +1,13 @@
+[Unit]
+Description=Filebeat
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+UMask=0027
+EnvironmentFile=/etc/default/filebeat
+ExecStart=/usr/bin/filebeat --environment systemd $FILEBEAT_OPTS
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
package/filebeat/filebeat.yml new file +5 -0
new file mode 100644
index 0000000000..147c1d55dd
@@ -0,0 +1,5 @@
+path:
+    home: /usr/share/filebeat
+    config: /etc/filebeat
+    data: /var/lib/filebeat
+    logs: /var/log/filebeat
Signature · 17 lines
2.34.1

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw
In reply to [email protected]

Hello,

+Cc Christian Steward.

Quoted 6 lines · On Thu, 9 Nov 2023 17:39:54 +0100 <[email protected]> wrote:

This two-patch series introduces filebeat, an ElasticSearch-related tool to grab local log files and send them to a remote ElasticSearch server.

It is a bit convoluted (see commit log), and requires a little extension in our golang-package infrastructure, which we introduce as the first patch in the series.

Christian, could you review this patch series, and especially PATCH 1/2, which touches the golang-package infrastructure?

Thanks a lot!

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

Hi Thomas,

Quoted 10 lines · Thomas Petazzoni

Hello,

+Cc Christian Steward.

Quoted 3 lines · On Thu, 9 Nov 2023 17:39:54 +0100 <[email protected]> wrote:

This two-patch series introduces filebeat, an ElasticSearch-related tool to grab local log files and send them to a remote ElasticSearch server.

Quoted 7 lines
Quoted 3 lines

It is a bit convoluted (see commit log), and requires a little extension in our golang-package infrastructure, which we introduce as the first patch in the series.

Christian, could you review this patch series, and especially PATCH 1/2, which touches the golang-package infrastructure?

This one looks good to me:

https://patchwork.ozlabs.org/project/buildroot/patch/876f3a7bb6a2375193fc8f06ab856d2449f83727.1699547993.git.yann.morin@orange.com/

As does the other, but I haven't tested it / built it. It looks fine from an initial review.

https://patchwork.ozlabs.org/project/buildroot/patch/46f20e86fa6c17ada33ede672ffe4229d8bf26a3.1699547993.git.yann.morin@orange.com/

Best regards,
Christian Stewart

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot