Journald fixes
cover Journald fixes
This patchset does two things. Enables systemd-journald-gatewayd Fixes journald for various memory leaks and memory allocation problems as seen under stress and big-endian architectures.
The following changes since commit b5b4898cd409036161c62891e9618d9ab3f891f9:
python-pycurl: upgrade to 7.19.3 (2014-02-08 21:03:27 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib kraj/journald-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kraj/journald-fixes
Khem Raj (3):
systemd: Fix misc journald memory bugs
libmicrohttpd: add recipe
systemd: Enable building microhttpd
...ournal-Add-missing-byte-order-conversions.patch | 60 ++++++++++++++ .../journal-file-protect-against-alloca-0.patch | 23 ++++++ ...n-appending-to-journal-file-allocate-larg.patch | 83 ++++++++++++++++++++ .../systemd/journald-add-missing-error-check.patch | 16 ++++ .../systemd/journald-fix-minor-memory-leak.patch | 16 ++++ meta/recipes-core/systemd/systemd_208.bb | 7 ++ .../libmicrohttpd/libmicrohttpd_0.9.33.bb | 18 +++++ 7 files changed, 223 insertions(+) create mode 100644 meta/recipes-core/systemd/systemd/journal-Add-missing-byte-order-conversions.patch create mode 100644 meta/recipes-core/systemd/systemd/journal-file-protect-against-alloca-0.patch create mode 100644 meta/recipes-core/systemd/systemd/journal-when-appending-to-journal-file-allocate-larg.patch create mode 100644 meta/recipes-core/systemd/systemd/journald-add-missing-error-check.patch create mode 100644 meta/recipes-core/systemd/systemd/journald-fix-minor-memory-leak.patch create mode 100644 meta/recipes-support/libmicrohttpd/libmicrohttpd_0.9.33.bb
Signature · 1 line
1.7.10.4
PATCH 1/3 systemd: Fix misc journald memory bugs
These set of patches fix journald exhibiting some issues under load.
One of the prevelant issues is that when appending to journal it is not able to allocate memory and starts taking 100% cpu spewing errors like
systemd-journald[2934]: Failed to write entry (19 items, 452 bytes), ignoring: Cannot allocate memory
Other memory issues crept up with time e.g.vacuuming
--- ...ournal-Add-missing-byte-order-conversions.patch | 60 ++++++++++++++ .../journal-file-protect-against-alloca-0.patch | 23 ++++++ ...n-appending-to-journal-file-allocate-larg.patch | 83 ++++++++++++++++++++ .../systemd/journald-add-missing-error-check.patch | 16 ++++ .../systemd/journald-fix-minor-memory-leak.patch | 16 ++++ meta/recipes-core/systemd/systemd_208.bb | 5 ++ 6 files changed, 203 insertions(+) create mode 100644 meta/recipes-core/systemd/systemd/journal-Add-missing-byte-order-conversions.patch create mode 100644 meta/recipes-core/systemd/systemd/journal-file-protect-against-alloca-0.patch create mode 100644 meta/recipes-core/systemd/systemd/journal-when-appending-to-journal-file-allocate-larg.patch create mode 100644 meta/recipes-core/systemd/systemd/journald-add-missing-error-check.patch create mode 100644 meta/recipes-core/systemd/systemd/journald-fix-minor-memory-leak.patch
meta/recipes-core/systemd/systemd/journal-Add-missing-byte-order-conversions.patch new file +60 -0
@@ -0,0 +1,60 @@ +From 43539d6b60ef0db3e98d00bef0024614c8c1807a Mon Sep 17 00:00:00 2001 +From: George McCollister <[email protected]> +Date: Tue, 31 Dec 2013 14:37:32 -0600 +Subject: [PATCH] journal: Add missing byte order conversions + +Convert entry_array.items[0] to host byte order prior to passing it to +chain_cache_put(). + +[zj: also use le64toh in journal-verify.c] + +https://bugs.freedesktop.org/show_bug.cgi?id=73194 + +Upstream-Status: Backport [Fedora] +--- + src/journal/journal-file.c | 4 ++-- + src/journal/journal-verify.c | 6 +++--- + 2 files changed, 5 insertions(+), 5 deletions(-) + +Index: systemd-208/src/journal/journal-file.c +=================================================================== +--- systemd-208.orig/src/journal/journal-file.c 2014-02-07 22:51:44.000000000 -0800 ++++ systemd-208/src/journal/journal-file.c 2014-02-07 22:58:40.665062951 -0800 +@@ -1447,7 +1447,7 @@ + + found: + /* Let's cache this item for the next invocation */ +- chain_cache_put(f->chain_cache, ci, first, a, o->entry_array.items[0], t); ++ chain_cache_put(f->chain_cache, ci, first, a, le64toh(o->entry_array.items[0]), t); + + r = journal_file_move_to_object(f, OBJECT_ENTRY, p, &o); + if (r < 0) +@@ -1624,7 +1624,7 @@ + return 0; + + /* Let's cache this item for the next invocation */ +- chain_cache_put(f->chain_cache, ci, first, a, array->entry_array.items[0], t); ++ chain_cache_put(f->chain_cache, ci, first, a, le64toh(array->entry_array.items[0]), t); + + if (subtract_one && i == 0) + p = last_p; +Index: systemd-208/src/journal/journal-verify.c +=================================================================== +--- systemd-208.orig/src/journal/journal-verify.c 2013-08-13 13:02:46.000000000 -0700 ++++ systemd-208/src/journal/journal-verify.c 2014-02-07 22:57:14.849308409 -0800 +@@ -249,12 +249,12 @@ + } + + for (i = 0; i < journal_file_entry_array_n_items(o); i++) +- if (o->entry_array.items[i] != 0 && +- !VALID64(o->entry_array.items[i])) { ++ if (le64toh(o->entry_array.items[i]) != 0 && ++ !VALID64(le64toh(o->entry_array.items[i]))) { + log_error(OFSfmt": invalid object entry array item (%"PRIu64"/%"PRIu64"): "OFSfmt, + offset, + i, journal_file_entry_array_n_items(o), +- o->entry_array.items[i]); ++ le64toh(o->entry_array.items[i])); + return -EBADMSG; + } +
meta/recipes-core/systemd/systemd/journal-file-protect-against-alloca-0.patch new file +23 -0
@@ -0,0 +1,23 @@ +From a25fd0d4bd3cf652e55c24e7dc873fe530fa111a Mon Sep 17 00:00:00 2001 +From: Thomas Hindoe Paaboel Andersen <[email protected]> +Date: Mon, 16 Dec 2013 23:35:30 +0100 +Subject: [PATCH] journal-file: protect against alloca(0) + +--- + src/journal/journal-file.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/src/journal/journal-file.c b/src/journal/journal-file.c +index 090cf97..8ea258b 100644 +--- a/src/journal/journal-file.c ++++ b/src/journal/journal-file.c +@@ -2737,7 +2737,8 @@ int journal_file_copy_entry(JournalFile *from, JournalFile *to, Object *o, uint6 + ts.realtime = le64toh(o->entry.realtime); + + n = journal_file_entry_n_items(o); +- items = alloca(sizeof(EntryItem) * n); ++ /* alloca() can't take 0, hence let's allocate at least one */ ++ items = alloca(sizeof(EntryItem) * MAX(1u, n)); + + for (i = 0; i < n; i++) { + uint64_t l, h;
meta/recipes-core/systemd/systemd/journal-when-appending-to-journal-file-allocate-larg.patch new file +83 -0
83 lines not on this page · raw message
meta/recipes-core/systemd/systemd/journald-add-missing-error-check.patch new file +16 -0
@@ -0,0 +1,16 @@ +Upstream-Status: Backport [Fedora] + +Index: systemd-208/src/journal/journal-file.c +=================================================================== +--- systemd-208.orig/src/journal/journal-file.c 2013-08-13 13:02:46.397707086 -0700 ++++ systemd-208/src/journal/journal-file.c 2014-02-07 22:29:01.398794277 -0800 +@@ -907,7 +907,8 @@ + + osize = offsetof(Object, field.payload) + size; + r = journal_file_append_object(f, OBJECT_FIELD, osize, &o, &p); +- ++ if (r < 0) ++ return r; + o->field.hash = htole64(hash); + memcpy(o->field.payload, field, size); +
meta/recipes-core/systemd/systemd/journald-fix-minor-memory-leak.patch new file +16 -0
@@ -0,0 +1,16 @@ +Fix minor memory leak + +Upstream-Status: Backport [Fedora] + +Index: systemd-208/src/journal/journal-vacuum.c +=================================================================== +--- systemd-208.orig/src/journal/journal-vacuum.c 2013-09-12 05:51:57.258256643 -0700 ++++ systemd-208/src/journal/journal-vacuum.c 2014-02-07 22:35:55.695747001 -0800 +@@ -277,6 +277,7 @@ + freed += size; + } else if (errno != ENOENT) + log_warning("Failed to delete %s/%s: %m", directory, p); ++ free(p); + + continue; + }
meta/recipes-core/systemd/systemd_208.bb +5 -0
@@ -20,6 +20,11 @@ inherit gtk-doc useradd pkgconfig autotools perlnative update-rc.d update-altern SRC_URI = "http://www.freedesktop.org/software/systemd/systemd-${PV}.tar.xz \ file://0001-Use-bin-mkdir-instead-of-host-mkdir-path.patch \ file://binfmt-install.patch \ + file://journald-add-missing-error-check.patch \ + file://journald-fix-minor-memory-leak.patch \ + file://journal-when-appending-to-journal-file-allocate-larg.patch \ + file://journal-file-protect-against-alloca-0.patch \ + file://journal-Add-missing-byte-order-conversions.patch \ file://touchscreen.rules \ ${UCLIBCPATCHES} \ file://00-create-volatile.conf \
Signature · 1 line
1.7.10.4
PATCH 2/3 libmicrohttpd: add recipe
libmicrohttpd is needed by systemd-journald-gatewayd
--- .../libmicrohttpd/libmicrohttpd_0.9.33.bb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 meta/recipes-support/libmicrohttpd/libmicrohttpd_0.9.33.bb
meta/recipes-support/libmicrohttpd/libmicrohttpd_0.9.33.bb new file +18 -0
@@ -0,0 +1,18 @@ +# Copyright (C) 2014 Khem Raj <[email protected]> +# Released under the MIT license (see COPYING.MIT for the terms) + +DESCRIPTION = "A small C library that is supposed to make it easy to run an HTTP server as part of another application" +HOMEPAGE = "http://www.gnu.org/software/libmicrohttpd/" +LICENSE = "LGPL-2.1+" +LIC_FILES_CHKSUM = "file://COPYING;md5=9331186f4f80db7da0e724bdd6554ee5" +SECTION = "net" +DEPENDS = "gnutls libgcrypt" + +SRC_URI = "http://ftp.gnu.org/gnu/libmicrohttpd/${BPN}-${PV}.tar.gz" +SRC_URI[md5sum] = "013b10f9de1cda5448b29c81305354a3" +SRC_URI[sha256sum] = "d9587e8497548f820650cd607daf3615ea3272435779378983032c570f1cd559" + +inherit autotools +EXTRA_OECONF += "--disable-dependency-tracking --enable-largefile \ + --enable-curl --enable-messages --with-pic" +
Signature · 1 line
1.7.10.4
PATCH 3/3 systemd: Enable building microhttpd
In reply to Khem RajThis enabled journald-gatewayd
--- meta/recipes-core/systemd/systemd_208.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-core/systemd/systemd_208.bb +2 -0
@@ -58,6 +58,7 @@ DEPENDS += "libgcrypt" PACKAGECONFIG[xz] = "--enable-xz,--disable-xz,xz" PACKAGECONFIG[tcp-wrappers] = "--enable-tcpwrap,--disable-tcpwrap,tcp-wrappers" PACKAGECONFIG[cryptsetup] = "--enable-libcryptsetup,--disable-libcryptsetup,cryptsetup" +PACKAGECONFIG[microhttpd] = "--enable-microhttpd,,libmicrohttpd" CACHED_CONFIGUREVARS = "ac_cv_path_KILL=${base_bindir}/kill"
@@ -149,6 +150,7 @@ SYSTEMD_PACKAGES = "${PN}-binfmt" SYSTEMD_SERVICE_${PN}-binfmt = "systemd-binfmt.service" USERADD_PACKAGES = "${PN}" +USERADD_PARAM_${PN} += "--system systemd-journal-gateway" GROUPADD_PARAM_${PN} = "-r lock; -r systemd-journal" FILES_${PN}-analyze = "${bindir}/systemd-analyze"
Signature · 1 line
1.7.10.4
Re: [PATCH 3/3] systemd: Enable building microhttpd
Quoted 1 line · Khem Raj
+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,,libmicrohttpd"
There isn't a disabled option here, and you don't set PACKAGECONFIG to enable it so in theory it's disabled by default. Is this sufficiently "core" to be in oe-core? Alternatively we could have the packageconfig blocks in oe-core but libmicrohttpd in meta-oe.
Ross
Re: [PATCH 3/3] systemd: Enable building microhttpd
Quoted 4 lines · Ross
Quoted 1 line · Khem Raj
+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,,libmicrohttpd"
There isn't a disabled option here,
thats the default one can add --disable-microhttpd too
and you don't set PACKAGECONFIG to
Quoted 3 lines
enable it so in theory it's disabled by default. Is this sufficiently "core" to be in oe-core? Alternatively we could have the packageconfig blocks in oe-core but libmicrohttpd in meta-oe.
I am fine either way. It just makes life harder for folks who don't use meta-oe but hopefully there are handful of them. I can resend remainder of patch to meta-oe
Quoted 2 lines
Ross
Re: [PATCH 3/3] systemd: Enable building microhttpd
Quoted 7 lines · Khem Raj
Quoted 4 lines · Ross
Quoted 1 line · Khem Raj
+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,,libmicrohttpd"
There isn't a disabled option here,
thats the default one can add --disable-microhttpd too
Please add it for clarity.
Quoted 8 lines
and you don't set PACKAGECONFIG to
Quoted 3 lines
enable it so in theory it's disabled by default. Is this sufficiently "core" to be in oe-core? Alternatively we could have the packageconfig blocks in oe-core but libmicrohttpd in meta-oe.
I am fine either way. It just makes life harder for folks who don't use meta-oe but hopefully there are handful of them. I can resend remainder of patch to meta-oe
This was just the usual "is it core?" question for new recipes. Anyone else have an opinion?
Ross
Re: [PATCH 3/3] systemd: Enable building microhttpd
Quoted 3 lines · Ross
Quoted 1 line
thats the default one can add --disable-microhttpd too
Please add it for clarity.
Do you suggest to add it to configure cmdline or in comments
Re: [PATCH 3/3] systemd: Enable building microhttpd
Quoted 6 lines · Khem Raj
Quoted 3 lines · Ross
Quoted 1 line
thats the default one can add --disable-microhttpd too
Please add it for clarity.
Do you suggest to add it to configure cmdline or in comments
I meant this:
+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,--disable-microhttpd,libmicrohttpd"
Clear to the person reading the .bb, and if systemd changes the behaviour when the option isn't specified (to enable, or automatically detect) then our build doesn't change.
Ross
Re: [PATCH 3/3] systemd: Enable building microhttpd
Quoted 4 lines · Ross
Quoted 3 lines · Khem Raj
Quoted 2 lines · Ross
Quoted 1 line
thats the default one can add --disable-microhttpd too
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Please add it for clarity.
Do you suggest to add it to configure cmdline or in comments
I meant this:
+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,--disable-microhttpd,libmicrohttpd"
We already had --disable-microhttpd in EXTRA_OECONF so I removed it from there and moved that piece to PACKAGECONFIG and pushed the patch to pull tree, take a look.
Quoted 6 lines
Clear to the person reading the .bb, and if systemd changes the behaviour when the option isn't specified (to enable, or automatically detect) then our build doesn't change.
Ross
Re: [PATCH 3/3] systemd: Enable building microhttpd
Quoted 3 lines · Khem Raj
We already had --disable-microhttpd in
EXTRA_OECONFso I removed it from there and moved that piece to PACKAGECONFIG and pushed the patch to pull tree, take a look.
Yeah, that. :)
Ross