Skip to contentSkip to conversation

syslog: fix sporadic failures due to rate-limitting

PATCH New Jan Stancek 2 participants 5 messages Jul 24, 2019 – Aug 6, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 syslog: fix sporadic failures due to rate-limitting

Some distros (Fedora30) do not have rsyslog installed by default. Also some test environments generate a lot of messages before they start syslogs tests.

This creates a problem if rsyslog is using imjournal, because by default rate-limitting is enabled. If there is a big backlog of messages prior to running syslog tests, some messages from tests may be dropped.

Easy way to reproduce is to stop rsyslog, delete imjournal.state, run kmsg01 test couple times and then run some syslog test:

  Jul 23 15:05:13 ... begin to drop messages due to rate-limiting
  Jul 23 15:05:21 ... 47006 messages lost due to rate-limiting

Messages make it to systemd journal, but not to log configured in rsyslog.

This patch adds Ratelimit.Interval="0" and IgnorePreviousMessages="on" to config for imjournal:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html

---
 testcases/kernel/syscalls/syslog/syslog-lib.sh | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
testcases/kernel/syscalls/syslog/syslog-lib.sh +12 -12
index 31266f93e729..eed501d9b3ad 100755
@@ -75,19 +75,19 @@ setup()
 		#   $ModLoad imjournal
 		#   module(load="imjournal"...)
 		# in rsyslog config, and using those settings.
-		if grep -qri '^[^#].*load.*imjournal' /etc/rsyslog.conf /etc/rsyslog.d/ ; then
-			systemd_journal=$(grep -Ehoi "^[^#].*(imjournal|workdirectory).*" -r /etc/rsyslog.conf /etc/rsyslog.d/)
-			RSYSLOG_CONFIG=$(cat <<EOF
-$systemd_journal
-EOF
-)
+		if grep -qri '^[^#]*modload.*imjournal' /etc/rsyslog.conf /etc/rsyslog.d/; then
+			RSYSLOG_CONFIG=$(grep -Ehoi "^[^#].*(imjournal|workdirectory).*" -r /etc/rsyslog.conf /etc/rsyslog.d/;
+				echo '$imjournalRatelimitInterval 0'; \
+				echo '$ImjournalIgnorePreviousMessages on';)
+		elif grep -qri '^[^#]*module.*load="imjournal"' /etc/rsyslog.conf /etc/rsyslog.d/; then
+			RSYSLOG_CONFIG=$(grep -Ehoi "^[^#].*workdirectory.*" -r /etc/rsyslog.conf /etc/rsyslog.d/; \
+				echo 'module(load="imjournal"'; \
+				echo '       StateFile="imjournal.state"'; \
+				echo '       Ratelimit.Interval="0"'; \
+				echo '       IgnorePreviousMessages="on")')
 		else
-			log_socket=$(grep -ho "^\$SystemLogSocketName .*" -r /etc/rsyslog.conf /etc/rsyslog.d/ | head -1)
-			RSYSLOG_CONFIG=$(cat <<EOF
-\$ModLoad imuxsock.so
-$log_socket
-EOF
-)
+			RSYSLOG_CONFIG=$(echo '$ModLoad imuxsock.so'; \
+				grep -ho "^\$SystemLogSocketName .*" -r /etc/rsyslog.conf /etc/rsyslog.d/ | head -1)
 		fi
 	else
 		tst_resm TCONF "Couldn't find syslogd, syslog-ng or rsyslogd"
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/1 syslog: fix sporadic failures due to rate-limitting

Hi Jan,

Quoted 8 lines · Jan Stancek

Some distros (Fedora30) do not have rsyslog installed by default. Also some test environments generate a lot of messages before they start syslogs tests.

This creates a problem if rsyslog is using imjournal, because by default rate-limitting is enabled. If there is a big backlog of messages prior to running syslog tests, some messages from tests may be dropped.

I'm not sure if I understand this situation, from your description, if the rsyslog is not installed,
then the /etc/rsyslog.conf will not exist, so how does the rsyslog using imjournal to effect on the test cases? Shouldn't it choose syslog or syslog-ng for testing?

if [ "$SYSLOG_DAEMON" = "syslog" ]; then
CONFIG_FILE="/etc/syslog.conf"
elif [ "$SYSLOG_DAEMON" = "syslog-ng" ]; then
CONFIG_FILE="/etc/syslog-ng/syslog-ng.conf"
elif [ "$SYSLOG_DAEMON" = "rsyslog" ]; then

        ....

fi

And from my test, if I remove(#rpm -e rsyslog) the rsyslog from system, this test will choose to go syslog-ng(/etc/syslog-ng/syslog-ng.conf) way to test. unfoutananly that always failed as:

--------------------------

syslog02    0  TINFO  :  Test if messages of all levels are logged.
syslog02    0  TINFO  :  For each level, a separate configuration file is
syslog02    0  TINFO  :  created and that will be used as syslog.conf file.
syslog02    0  TINFO  :  testing whether messages are logged into log file
syslog02    0  TINFO  :  Doing level: emerg...
syslog02    0  TINFO  :  restarting syslog daemon
Broadcast message from
[email protected] (Tue
2019-07-30 09:42:24 EDT):
syslogtst[26081]: syslogtst: mail emerg test.
syslog02    1  TFAIL  :  ltpapicmd.c:188: ***** Level emerg failed *****
syslog02    0  TINFO  :  Doing level: alert...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    2  TFAIL  :  ltpapicmd.c:188: ***** Level alert failed *****
syslog02    0  TINFO  :  Doing level: crit...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    3  TFAIL  :  ltpapicmd.c:188: ***** Level crit failed *****
syslog02    0  TINFO  :  Doing level: err...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    4  TFAIL  :  ltpapicmd.c:188: ***** Level err failed *****
syslog02    0  TINFO  :  Doing level: warning...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    5  TFAIL  :  ltpapicmd.c:188: ***** Level warning failed *****
syslog02    0  TINFO  :  Doing level: notice...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    6  TFAIL  :  ltpapicmd.c:188: ***** Level notice failed *****
syslog02    0  TINFO  :  Doing level: info...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    7  TFAIL  :  ltpapicmd.c:188: ***** Level info failed *****
syslog02    0  TINFO  :  Doing level: debug...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    8  TFAIL  :  ltpapicmd.c:188: ***** Level debug failed *****
syslog02    0  TINFO  :  restarting syslog daemon
Quoted 5 lines

Easy way to reproduce is to stop rsyslog, delete imjournal.state, run kmsg01 test couple times and then run some syslog test:

  Jul 23 15:05:13 ... begin to drop messages due to rate-limiting
  Jul 23 15:05:21 ... 47006 messages lost due to rate-limiting

Messages make it to systemd journal, but not to log configured in rsyslog.

I didn't reproduce this problem, if I stop rsyslog(systemctl stop rsyslog) by manual, the test will enable it automatically:

---------------------
syslog02 0 TINFO : restarting syslog daemon
...
syslogtst[26848]: syslogtst: mail emerg test.
syslog02 1 TPASS : ***** Level emerg passed *****
------------------------

or maybe I missed something?

--
Regards,
Li Wang

#2 lore raw

PATCH 1/1 syslog: fix sporadic failures due to rate-limitting

----- Original Message -----

Quoted 8 lines

Hi Jan,

Quoted 5 lines · Jan Stancek

Some distros (Fedora30) do not have rsyslog installed by default. Also some test environments generate a lot of messages before they start syslogs tests.

Quoted 6 lines
Quoted 3 lines

This creates a problem if rsyslog is using imjournal, because by default rate-limitting is enabled. If there is a big backlog of messages prior to running syslog tests, some messages from tests may be dropped.

I'm not sure if I understand this situation, from your description, if the rsyslog is not installed,

OK, that might be specific to our harness. We install rsyslog before LTP runs. My point was that rsyslog wasn't running when system started, and therefore the backlog kept on growing.

Quoted 61 lines

then the /etc/rsyslog.conf will not exist, so how does the rsyslog using imjournal to effect on the test cases? Shouldn't it choose syslog or syslog-ng for testing?

if [ "$SYSLOG_DAEMON" = "syslog" ]; then
CONFIG_FILE="/etc/syslog.conf"
elif [ "$SYSLOG_DAEMON" = "syslog-ng" ]; then
CONFIG_FILE="/etc/syslog-ng/syslog-ng.conf"
elif [ "$SYSLOG_DAEMON" = "rsyslog" ]; then

        ....

fi

And from my test, if I remove(#rpm -e rsyslog) the rsyslog from system, this test will choose to go syslog-ng(/etc/syslog-ng/syslog-ng.conf) way to test. unfoutananly that always failed as:

--------------------------

syslog02    0  TINFO  :  Test if messages of all levels are logged.
syslog02    0  TINFO  :  For each level, a separate configuration file is
syslog02    0  TINFO  :  created and that will be used as syslog.conf file.
syslog02    0  TINFO  :  testing whether messages are logged into log file
syslog02    0  TINFO  :  Doing level: emerg...
syslog02    0  TINFO  :  restarting syslog daemon
Broadcast message from
[email protected] (Tue
2019-07-30 09:42:24 EDT):
syslogtst[26081]: syslogtst: mail emerg test.
syslog02    1  TFAIL  :  ltpapicmd.c:188: ***** Level emerg failed *****
syslog02    0  TINFO  :  Doing level: alert...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    2  TFAIL  :  ltpapicmd.c:188: ***** Level alert failed *****
syslog02    0  TINFO  :  Doing level: crit...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    3  TFAIL  :  ltpapicmd.c:188: ***** Level crit failed *****
syslog02    0  TINFO  :  Doing level: err...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    4  TFAIL  :  ltpapicmd.c:188: ***** Level err failed *****
syslog02    0  TINFO  :  Doing level: warning...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    5  TFAIL  :  ltpapicmd.c:188: ***** Level warning failed *****
syslog02    0  TINFO  :  Doing level: notice...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    6  TFAIL  :  ltpapicmd.c:188: ***** Level notice failed *****
syslog02    0  TINFO  :  Doing level: info...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    7  TFAIL  :  ltpapicmd.c:188: ***** Level info failed *****
syslog02    0  TINFO  :  Doing level: debug...
syslog02    0  TINFO  :  restarting syslog daemon
syslog02    8  TFAIL  :  ltpapicmd.c:188: ***** Level debug failed *****
syslog02    0  TINFO  :  restarting syslog daemon
Quoted 5 lines

Easy way to reproduce is to stop rsyslog, delete imjournal.state, run kmsg01 test couple times and then run some syslog test:

  Jul 23 15:05:13 ... begin to drop messages due to rate-limiting
  Jul 23 15:05:21 ... 47006 messages lost due to rate-limiting

Messages make it to systemd journal, but not to log configured in rsyslog.

I didn't reproduce this problem, if I stop rsyslog(systemctl stop rsyslog) by manual, the test will enable it automatically:

Quoted 8 lines

---------------------
syslog02 0 TINFO : restarting syslog daemon
...
syslogtst[26848]: syslogtst: mail emerg test.
syslog02 1 TPASS : ***** Level emerg passed *****
------------------------

or maybe I missed something?

Have you followed all the steps?
- stop rsyslog
- delete imjournal.state
- run kmsg01 couple times
- run syslog01

Quoted 5 lines

--
Regards,
Li Wang

#3 lore raw

PATCH 1/1 syslog: fix sporadic failures due to rate-limitting

Hi Jan,

Quoted 5 lines

Have you followed all the steps?
- stop rsyslog
- delete imjournal.state
- run kmsg01 couple times
- run syslog01

Finally, I run more times for ksm01 to make a large backlog of messages then reproduced it. After applying your patch, the problem does not appear anymore. So, ACK for this patch.

---------- Reproduce Log -----------

# cat runtest/wangli
kmsg01 kmsg01
kmsg01 kmsg01
kmsg01 kmsg01
kmsg01 kmsg01
kmsg01 kmsg01
syslog01 syslog01
# systemctl stop rsyslog; rm -fr /var/lib/rsyslog/*; ./runltp -f wangli
syslog01 FAIL
# journalctl
...
Jul 31 04:53:18 hp-dl385pg8-11.rhts.eng.pek2.redhat.com
rsyslogd[3336]: imjournal from <hp-dl385pg8-11:restraintd>: begin to
drop messages due to rate-limiting
Jul 31 04:53:19 hp-dl385pg8-11.rhts.eng.pek2.redhat.com
syslogtst[3341]: syslogtst: mail info test.
Jul 31 04:53:21 hp-dl385pg8-11.rhts.eng.pek2.redhat.com systemd[1]:
Stopping System Logging Service...
Jul 31 04:53:21 hp-dl385pg8-11.rhts.eng.pek2.redhat.com
rsyslogd[3336]: imjournal: 63528 messages lost due to rate-limiting
Signature · 2 lines
Regards,
Li Wang
#4 lore raw

PATCH 1/1 syslog: fix sporadic failures due to rate-limitting

----- Original Message -----

Quoted 11 lines

Hi Jan,

Quoted 5 lines

Have you followed all the steps?
- stop rsyslog
- delete imjournal.state
- run kmsg01 couple times
- run syslog01

Finally, I run more times for ksm01 to make a large backlog of messages then reproduced it. After applying your patch, the problem does not appear anymore. So, ACK for this patch.

Pushed.

Thanks,
Jan