Skip to contentSkip to conversation

Journald fixes

PATCH In review Khem Raj 2 participants 11 messages Feb 11, 2014

lore
Share & tools
lore

Khem Raj

author
#0 lore raw

cover Journald fixes

This patchset does two things. Enables systemd-journald-gatewayd Fixes journald for various memory leaks and memory allocation problems as seen under stress and big-endian architectures.

The following changes since commit b5b4898cd409036161c62891e9618d9ab3f891f9:

python-pycurl: upgrade to 7.19.3 (2014-02-08 21:03:27 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib kraj/journald-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kraj/journald-fixes

Khem Raj (3):
systemd: Fix misc journald memory bugs
libmicrohttpd: add recipe
systemd: Enable building microhttpd

 ...ournal-Add-missing-byte-order-conversions.patch |   60 ++++++++++++++
 .../journal-file-protect-against-alloca-0.patch    |   23 ++++++
 ...n-appending-to-journal-file-allocate-larg.patch |   83 ++++++++++++++++++++
 .../systemd/journald-add-missing-error-check.patch |   16 ++++
 .../systemd/journald-fix-minor-memory-leak.patch   |   16 ++++
 meta/recipes-core/systemd/systemd_208.bb           |    7 ++
 .../libmicrohttpd/libmicrohttpd_0.9.33.bb          |   18 +++++
 7 files changed, 223 insertions(+)
 create mode 100644 meta/recipes-core/systemd/systemd/journal-Add-missing-byte-order-conversions.patch
 create mode 100644 meta/recipes-core/systemd/systemd/journal-file-protect-against-alloca-0.patch
 create mode 100644 meta/recipes-core/systemd/systemd/journal-when-appending-to-journal-file-allocate-larg.patch
 create mode 100644 meta/recipes-core/systemd/systemd/journald-add-missing-error-check.patch
 create mode 100644 meta/recipes-core/systemd/systemd/journald-fix-minor-memory-leak.patch
 create mode 100644 meta/recipes-support/libmicrohttpd/libmicrohttpd_0.9.33.bb
Signature · 1 line
1.7.10.4
author
#1 lore raw

PATCH 1/3 systemd: Fix misc journald memory bugs

These set of patches fix journald exhibiting some issues under load.

One of the prevelant issues is that when appending to journal it is not able to allocate memory and starts taking 100% cpu spewing errors like

systemd-journald[2934]: Failed to write entry (19 items, 452 bytes), ignoring: Cannot allocate memory

Other memory issues crept up with time e.g.vacuuming

---
 ...ournal-Add-missing-byte-order-conversions.patch |   60 ++++++++++++++
 .../journal-file-protect-against-alloca-0.patch    |   23 ++++++
 ...n-appending-to-journal-file-allocate-larg.patch |   83 ++++++++++++++++++++
 .../systemd/journald-add-missing-error-check.patch |   16 ++++
 .../systemd/journald-fix-minor-memory-leak.patch   |   16 ++++
 meta/recipes-core/systemd/systemd_208.bb           |    5 ++
 6 files changed, 203 insertions(+)
 create mode 100644 meta/recipes-core/systemd/systemd/journal-Add-missing-byte-order-conversions.patch
 create mode 100644 meta/recipes-core/systemd/systemd/journal-file-protect-against-alloca-0.patch
 create mode 100644 meta/recipes-core/systemd/systemd/journal-when-appending-to-journal-file-allocate-larg.patch
 create mode 100644 meta/recipes-core/systemd/systemd/journald-add-missing-error-check.patch
 create mode 100644 meta/recipes-core/systemd/systemd/journald-fix-minor-memory-leak.patch
meta/recipes-core/systemd/systemd/journal-Add-missing-byte-order-conversions.patch new file +60 -0
new file mode 100644
index 0000000..21ea0f9
@@ -0,0 +1,60 @@
+From 43539d6b60ef0db3e98d00bef0024614c8c1807a Mon Sep 17 00:00:00 2001
+From: George McCollister <[email protected]>
+Date: Tue, 31 Dec 2013 14:37:32 -0600
+Subject: [PATCH] journal: Add missing byte order conversions
+
+Convert entry_array.items[0] to host byte order prior to passing it to
+chain_cache_put().
+
+[zj: also use le64toh in journal-verify.c]
+
+https://bugs.freedesktop.org/show_bug.cgi?id=73194
+
+Upstream-Status: Backport [Fedora]
+---
+ src/journal/journal-file.c   | 4 ++--
+ src/journal/journal-verify.c | 6 +++---
+ 2 files changed, 5 insertions(+), 5 deletions(-)
+
+Index: systemd-208/src/journal/journal-file.c
+===================================================================
+--- systemd-208.orig/src/journal/journal-file.c	2014-02-07 22:51:44.000000000 -0800
++++ systemd-208/src/journal/journal-file.c	2014-02-07 22:58:40.665062951 -0800
+@@ -1447,7 +1447,7 @@
+ 
+ found:
+         /* Let's cache this item for the next invocation */
+-        chain_cache_put(f->chain_cache, ci, first, a, o->entry_array.items[0], t);
++        chain_cache_put(f->chain_cache, ci, first, a, le64toh(o->entry_array.items[0]), t);
+ 
+         r = journal_file_move_to_object(f, OBJECT_ENTRY, p, &o);
+         if (r < 0)
+@@ -1624,7 +1624,7 @@
+                 return 0;
+ 
+         /* Let's cache this item for the next invocation */
+-        chain_cache_put(f->chain_cache, ci, first, a, array->entry_array.items[0], t);
++        chain_cache_put(f->chain_cache, ci, first, a, le64toh(array->entry_array.items[0]), t);
+ 
+         if (subtract_one && i == 0)
+                 p = last_p;
+Index: systemd-208/src/journal/journal-verify.c
+===================================================================
+--- systemd-208.orig/src/journal/journal-verify.c	2013-08-13 13:02:46.000000000 -0700
++++ systemd-208/src/journal/journal-verify.c	2014-02-07 22:57:14.849308409 -0800
+@@ -249,12 +249,12 @@
+                 }
+ 
+                 for (i = 0; i < journal_file_entry_array_n_items(o); i++)
+-                        if (o->entry_array.items[i] != 0 &&
+-                            !VALID64(o->entry_array.items[i])) {
++                        if (le64toh(o->entry_array.items[i]) != 0 &&
++                            !VALID64(le64toh(o->entry_array.items[i]))) {
+                                 log_error(OFSfmt": invalid object entry array item (%"PRIu64"/%"PRIu64"): "OFSfmt,
+                                           offset,
+                                           i, journal_file_entry_array_n_items(o),
+-                                          o->entry_array.items[i]);
++                                          le64toh(o->entry_array.items[i]));
+                                 return -EBADMSG;
+                         }
+ 
meta/recipes-core/systemd/systemd/journal-file-protect-against-alloca-0.patch new file +23 -0
new file mode 100644
index 0000000..953373e
@@ -0,0 +1,23 @@
+From a25fd0d4bd3cf652e55c24e7dc873fe530fa111a Mon Sep 17 00:00:00 2001
+From: Thomas Hindoe Paaboel Andersen <[email protected]>
+Date: Mon, 16 Dec 2013 23:35:30 +0100
+Subject: [PATCH] journal-file: protect against alloca(0)
+
+---
+ src/journal/journal-file.c | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/src/journal/journal-file.c b/src/journal/journal-file.c
+index 090cf97..8ea258b 100644
+--- a/src/journal/journal-file.c
++++ b/src/journal/journal-file.c
+@@ -2737,7 +2737,8 @@ int journal_file_copy_entry(JournalFile *from, JournalFile *to, Object *o, uint6
+         ts.realtime = le64toh(o->entry.realtime);
+ 
+         n = journal_file_entry_n_items(o);
+-        items = alloca(sizeof(EntryItem) * n);
++        /* alloca() can't take 0, hence let's allocate at least one */
++        items = alloca(sizeof(EntryItem) * MAX(1u, n));
+ 
+         for (i = 0; i < n; i++) {
+                 uint64_t l, h;
meta/recipes-core/systemd/systemd/journal-when-appending-to-journal-file-allocate-larg.patch new file +83 -0

83 lines not on this page · raw message

meta/recipes-core/systemd/systemd/journald-add-missing-error-check.patch new file +16 -0
new file mode 100644
index 0000000..10590e1
@@ -0,0 +1,16 @@
+Upstream-Status: Backport [Fedora]
+
+Index: systemd-208/src/journal/journal-file.c
+===================================================================
+--- systemd-208.orig/src/journal/journal-file.c	2013-08-13 13:02:46.397707086 -0700
++++ systemd-208/src/journal/journal-file.c	2014-02-07 22:29:01.398794277 -0800
+@@ -907,7 +907,8 @@
+ 
+         osize = offsetof(Object, field.payload) + size;
+         r = journal_file_append_object(f, OBJECT_FIELD, osize, &o, &p);
+-
++	if (r < 0)
++		return r;
+         o->field.hash = htole64(hash);
+         memcpy(o->field.payload, field, size);
+ 
meta/recipes-core/systemd/systemd/journald-fix-minor-memory-leak.patch new file +16 -0
new file mode 100644
index 0000000..9b90ed2
@@ -0,0 +1,16 @@
+Fix minor memory leak
+
+Upstream-Status: Backport [Fedora]
+
+Index: systemd-208/src/journal/journal-vacuum.c
+===================================================================
+--- systemd-208.orig/src/journal/journal-vacuum.c	2013-09-12 05:51:57.258256643 -0700
++++ systemd-208/src/journal/journal-vacuum.c	2014-02-07 22:35:55.695747001 -0800
+@@ -277,6 +277,7 @@
+                                 freed += size;
+                         } else if (errno != ENOENT)
+                                 log_warning("Failed to delete %s/%s: %m", directory, p);
++			free(p);
+ 
+                         continue;
+                 }
meta/recipes-core/systemd/systemd_208.bb +5 -0
index 5ed31c3..f94ce56 100644
@@ -20,6 +20,11 @@ inherit gtk-doc useradd pkgconfig autotools perlnative update-rc.d update-altern
 SRC_URI = "http://www.freedesktop.org/software/systemd/systemd-${PV}.tar.xz \
            file://0001-Use-bin-mkdir-instead-of-host-mkdir-path.patch \
            file://binfmt-install.patch \
+           file://journald-add-missing-error-check.patch \
+           file://journald-fix-minor-memory-leak.patch \
+           file://journal-when-appending-to-journal-file-allocate-larg.patch \
+           file://journal-file-protect-against-alloca-0.patch \
+           file://journal-Add-missing-byte-order-conversions.patch \
            file://touchscreen.rules \
            ${UCLIBCPATCHES} \
            file://00-create-volatile.conf \
Signature · 1 line
1.7.10.4
author
#2 lore raw

PATCH 2/3 libmicrohttpd: add recipe

libmicrohttpd is needed by systemd-journald-gatewayd

---
 .../libmicrohttpd/libmicrohttpd_0.9.33.bb            |   18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
 create mode 100644 meta/recipes-support/libmicrohttpd/libmicrohttpd_0.9.33.bb
meta/recipes-support/libmicrohttpd/libmicrohttpd_0.9.33.bb new file +18 -0
new file mode 100644
index 0000000..d56af76
@@ -0,0 +1,18 @@
+# Copyright (C) 2014 Khem Raj <[email protected]>
+# Released under the MIT license (see COPYING.MIT for the terms)
+
+DESCRIPTION = "A small C library that is supposed to make it easy to run an HTTP server as part of another application"
+HOMEPAGE = "http://www.gnu.org/software/libmicrohttpd/"
+LICENSE = "LGPL-2.1+"
+LIC_FILES_CHKSUM = "file://COPYING;md5=9331186f4f80db7da0e724bdd6554ee5"
+SECTION = "net"
+DEPENDS = "gnutls libgcrypt"
+
+SRC_URI = "http://ftp.gnu.org/gnu/libmicrohttpd/${BPN}-${PV}.tar.gz"
+SRC_URI[md5sum] = "013b10f9de1cda5448b29c81305354a3"
+SRC_URI[sha256sum] = "d9587e8497548f820650cd607daf3615ea3272435779378983032c570f1cd559"
+
+inherit autotools
+EXTRA_OECONF += "--disable-dependency-tracking --enable-largefile \
+                 --enable-curl --enable-messages --with-pic"
+
Signature · 1 line
1.7.10.4
author
#3 lore raw

PATCH 3/3 systemd: Enable building microhttpd

In reply to Khem Raj

This enabled journald-gatewayd

---
 meta/recipes-core/systemd/systemd_208.bb |    2 ++
 1 file changed, 2 insertions(+)
meta/recipes-core/systemd/systemd_208.bb +2 -0
index f94ce56..fe88a7a 100644
@@ -58,6 +58,7 @@ DEPENDS += "libgcrypt"
 PACKAGECONFIG[xz] = "--enable-xz,--disable-xz,xz"
 PACKAGECONFIG[tcp-wrappers] = "--enable-tcpwrap,--disable-tcpwrap,tcp-wrappers"
 PACKAGECONFIG[cryptsetup] = "--enable-libcryptsetup,--disable-libcryptsetup,cryptsetup"
+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,,libmicrohttpd"
 
 CACHED_CONFIGUREVARS = "ac_cv_path_KILL=${base_bindir}/kill"
 
@@ -149,6 +150,7 @@ SYSTEMD_PACKAGES = "${PN}-binfmt"
 SYSTEMD_SERVICE_${PN}-binfmt = "systemd-binfmt.service"
 
 USERADD_PACKAGES = "${PN}"
+USERADD_PARAM_${PN} += "--system systemd-journal-gateway"
 GROUPADD_PARAM_${PN} = "-r lock; -r systemd-journal"
 
 FILES_${PN}-analyze = "${bindir}/systemd-analyze"
Signature · 1 line
1.7.10.4
#4 lore raw

Re: [PATCH 3/3] systemd: Enable building microhttpd

Quoted 1 line · Khem Raj

+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,,libmicrohttpd"

There isn't a disabled option here, and you don't set PACKAGECONFIG to enable it so in theory it's disabled by default. Is this sufficiently "core" to be in oe-core? Alternatively we could have the packageconfig blocks in oe-core but libmicrohttpd in meta-oe.

Ross

author
#5 lore raw

Re: [PATCH 3/3] systemd: Enable building microhttpd

Quoted 4 lines · Ross
Quoted 1 line · Khem Raj

+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,,libmicrohttpd"

There isn't a disabled option here,

thats the default one can add --disable-microhttpd too

and you don't set PACKAGECONFIG to

Quoted 3 lines

enable it so in theory it's disabled by default. Is this sufficiently "core" to be in oe-core? Alternatively we could have the packageconfig blocks in oe-core but libmicrohttpd in meta-oe.

I am fine either way. It just makes life harder for folks who don't use meta-oe but hopefully there are handful of them. I can resend remainder of patch to meta-oe

Quoted 2 lines

Ross

#6 lore raw

Re: [PATCH 3/3] systemd: Enable building microhttpd

Quoted 7 lines · Khem Raj
Quoted 4 lines · Ross
Quoted 1 line · Khem Raj

+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,,libmicrohttpd"

There isn't a disabled option here,

thats the default one can add --disable-microhttpd too

Please add it for clarity.

Quoted 8 lines

and you don't set PACKAGECONFIG to

Quoted 3 lines

enable it so in theory it's disabled by default. Is this sufficiently "core" to be in oe-core? Alternatively we could have the packageconfig blocks in oe-core but libmicrohttpd in meta-oe.

I am fine either way. It just makes life harder for folks who don't use meta-oe but hopefully there are handful of them. I can resend remainder of patch to meta-oe

This was just the usual "is it core?" question for new recipes. Anyone else have an opinion?

Ross

author
#7 lore raw

Re: [PATCH 3/3] systemd: Enable building microhttpd

Quoted 3 lines · Ross
Quoted 1 line

thats the default one can add --disable-microhttpd too

Please add it for clarity.

Do you suggest to add it to configure cmdline or in comments

#8 lore raw

Re: [PATCH 3/3] systemd: Enable building microhttpd

Quoted 6 lines · Khem Raj
Quoted 3 lines · Ross
Quoted 1 line

thats the default one can add --disable-microhttpd too

Please add it for clarity.

Do you suggest to add it to configure cmdline or in comments

I meant this:

+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,--disable-microhttpd,libmicrohttpd"

Clear to the person reading the .bb, and if systemd changes the behaviour when the option isn't specified (to enable, or automatically detect) then our build doesn't change.

Ross

author
#9 lore raw

Re: [PATCH 3/3] systemd: Enable building microhttpd

Quoted 4 lines · Ross
Quoted 3 lines · Khem Raj
Quoted 2 lines · Ross
Quoted 1 line

thats the default one can add --disable-microhttpd too

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Please add it for clarity.

Do you suggest to add it to configure cmdline or in comments

I meant this:

+PACKAGECONFIG[microhttpd] = "--enable-microhttpd,--disable-microhttpd,libmicrohttpd"

We already had --disable-microhttpd in EXTRA_OECONF so I removed it from there and moved that piece to PACKAGECONFIG and pushed the patch to pull tree, take a look.

Quoted 6 lines

Clear to the person reading the .bb, and if systemd changes the behaviour when the option isn't specified (to enable, or automatically detect) then our build doesn't change.

Ross

#10 lore raw

Re: [PATCH 3/3] systemd: Enable building microhttpd

Quoted 3 lines · Khem Raj

We already had --disable-microhttpd in EXTRA_OECONF so I removed it from there and moved that piece to PACKAGECONFIG and pushed the patch to pull tree, take a look.

Yeah, that. :)

Ross