3.0.48-rt72-rc1 stable review

Dear RT Folks,

This is the RT stable review cycle of patch 3.0.48-rt72-rc1.

Please scream at me if I messed something up. Please test the patches too.

The -rc release will be uploaded to kernel.org and will be deleted when the final release is out. This is just a review release (or release candidate).

The pre-releases will not be pushed to the git repository, only the final release is.

If all goes well, this patch will be converted to the next main release on 11/2/2012.

Enjoy,

-- Steve

To build 3.0.48-rt72-rc1 directly, the following patches should be applied:

http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.xz

http://www.kernel.org/pub/linux/kernel/v3.0/patch-3.0.48.xz

http://www.kernel.org/pub/linux/kernel/projects/rt/3.0/patch-3.0.48-rt72-rc1.patch.xz

You can also build from 3.0.48-rt71 by applying the incremental patch:

http://www.kernel.org/pub/linux/kernel/projects/rt/3.0/incr/patch-3.0.48-rt71-rt72-rc1.patch.xz

Changes from 3.0.48-rt71:

---

Steven Rostedt (1):

      Linux 3.0.48-rt72-rc1

Thomas Gleixner (2):
rcu: Disable RCU_FAST_NO_HZ on RT
net: netfilter: Serialize xt_write_recseq sections on RT

Watanabe (1):
hrtimer: Raise softirq if hrtimer irq stalled

----

 include/linux/locallock.h          |    4 ++++
 include/linux/netfilter/x_tables.h |    7 +++++++
 init/Kconfig                       |    2 +-
 kernel/hrtimer.c                   |    9 ++++-----
 localversion-rt                    |    2 +-
 net/netfilter/core.c               |    6 ++++++
 6 files changed, 23 insertions(+), 7 deletions(-)

From: Watanabe <[email protected]>

When the hrtimer stall detection hits the softirq is not raised.

---
 kernel/hrtimer.c |    9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

index 7327846..87b4917 100644
@@ -1527,11 +1527,7 @@ retry:
 	if (expires_next.tv64 == KTIME_MAX ||
 	    !tick_program_event(expires_next, 0)) {
 		cpu_base->hang_detected = 0;
-
-		if (raise)
-			raise_softirq_irqoff(HRTIMER_SOFTIRQ);
-
-		return;
+		goto out;
 	}
 
 	/*

@@ -1575,6 +1571,9 @@ retry:
 	tick_program_event(expires_next, 1);
 	printk_once(KERN_WARNING "hrtimer: interrupt took %llu ns\n",
 		    ktime_to_ns(delta));
+out:
+	if (raise)
+		raise_softirq_irqoff(HRTIMER_SOFTIRQ);
 }
 
 /*

1.7.10.4

From: Thomas Gleixner <[email protected]>

This uses a timer_list timer from the irq disabled guts of the idle code. Disable it for now to prevent wreckage.

---
 init/Kconfig |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 5ed453f..948350f 100644
@@ -469,7 +469,7 @@ config RCU_FANOUT_EXACT
 
 config RCU_FAST_NO_HZ
 	bool "Accelerate last non-dyntick-idle CPU's grace periods"
-	depends on TREE_RCU && NO_HZ && SMP
+	depends on TREE_RCU && NO_HZ && SMP && !PREEMPT_RT_FULL
 	default n
 	help
 	  This option causes RCU to attempt to accelerate grace periods

1.7.10.4

From: Thomas Gleixner <[email protected]>

The netfilter code relies only on the implicit semantics of local_bh_disable() for serializing wt_write_recseq sections. RT breaks that and needs explicit serialization here.

---
 include/linux/locallock.h          |    4 ++++
 include/linux/netfilter/x_tables.h |    7 +++++++
 net/netfilter/core.c               |    6 ++++++
 3 files changed, 17 insertions(+)

index f1804a3..a5eea5d 100644
@@ -25,6 +25,9 @@ struct local_irq_lock {
 	DEFINE_PER_CPU(struct local_irq_lock, lvar) = {			\
 		.lock = __SPIN_LOCK_UNLOCKED((lvar).lock) }
 
+#define DECLARE_LOCAL_IRQ_LOCK(lvar)					\
+	DECLARE_PER_CPU(struct local_irq_lock, lvar)
+
 #define local_irq_lock_init(lvar)					\
 	do {								\
 		int __cpu;						\

@@ -220,6 +223,7 @@ static inline int __local_unlock_irqrestore(struct local_irq_lock *lv,
 #else /* PREEMPT_RT_BASE */
 
 #define DEFINE_LOCAL_IRQ_LOCK(lvar)		__typeof__(const int) lvar
+#define DECLARE_LOCAL_IRQ_LOCK(lvar)		extern __typeof__(const int) lvar
 
 static inline void local_irq_lock_init(int lvar) { }
 

index 32cddf7..bed90da2 100644
@@ -186,6 +186,7 @@ struct xt_counters_info {
 #ifdef __KERNEL__
 
 #include <linux/netdevice.h>
+#include <linux/locallock.h>
 
 /**
  * struct xt_action_param - parameters for matches/targets

@@ -466,6 +467,8 @@ extern void xt_free_table_info(struct xt_table_info *info);
  */
 DECLARE_PER_CPU(seqcount_t, xt_recseq);
 
+DECLARE_LOCAL_IRQ_LOCK(xt_write_lock);
+
 /**
  * xt_write_recseq_begin - start of a write section
  *

@@ -480,6 +483,9 @@ static inline unsigned int xt_write_recseq_begin(void)
 {
 	unsigned int addend;
 
+	/* RT protection */
+	local_lock(xt_write_lock);
+
 	/*
 	 * Low order bit of sequence is set if we already
 	 * called xt_write_recseq_begin().

@@ -510,6 +516,7 @@ static inline void xt_write_recseq_end(unsigned int addend)
 	/* this is kind of a write_seqcount_end(), but addend is 0 or 1 */
 	smp_wmb();
 	__this_cpu_add(xt_recseq.sequence, addend);
+	local_unlock(xt_write_lock);
 }
 
 /*

index 899b71c..5db16ea 100644
@@ -20,11 +20,17 @@
 #include <linux/proc_fs.h>
 #include <linux/mutex.h>
 #include <linux/slab.h>
+#include <linux/locallock.h>
 #include <net/net_namespace.h>
 #include <net/sock.h>
 
 #include "nf_internals.h"
 
+#ifdef CONFIG_PREEMPT_RT_BASE
+DEFINE_LOCAL_IRQ_LOCK(xt_write_lock);
+EXPORT_PER_CPU_SYMBOL(xt_write_lock);
+#endif
+
 static DEFINE_MUTEX(afinfo_mutex);
 
 const struct nf_afinfo __rcu *nf_afinfo[NFPROTO_NUMPROTO] __read_mostly;

1.7.10.4

Cough. You are missing a boat load of crucial fixes. There is a damned good reason why 3.0.stable got 12 updates and the -rt version 14.

Your risk assessment is definitley interesting.

Thanks,

tglx

I don't doubt there are. But we've only experienced one problem between 3.0.36-rt58 and 3.0.48-rt72. Indeed, it might be easier to evaluate the risk if all the changelogs were readily available. So far, however, we haven't discovered any other issues that give us any concerns.

We have specifically tested 3.0.36-rt58 for performance in our application. Moving to 3.0.48-rt72 poses risk about whether would still be able to meet our performance requirements. And with regard to stability, the only issue we've had since our earlier moved to 3.0.36-rt58 was the netfilter problem. We initially only saw the problem on PPP related interfaces, so we reverted to 3.0.3 (non-rt) for PPP related issues.

Now, if you think there are some issues that should raise eyebrows, or can point me to a list of changelogs, I'd be happy to evaluate them.

Thanks,
Pete

They are, it's just many of them. This is the point of the stable series, is to add patches that pose little risk and fix a real bug. Also note that a lot of the changes in mainline stable are for drivers, which most likely wont affect you. You may want to look at all the core code changes though.

Note, if you are only interested in -rt related fixes, all -rt only fixes are pushed as a separate update. Look for releases where only the -rt version changes. I update the -rt version when doing a new merge of the mainline stable release and I update the -rt version if there's -rt only changes added. I never combine the two, so you don't need to look at the changes where the mainline version number changes (ie. 3.0.x).

v3.0.34-rt55
v3.0.35-rt56
v3.0.36-rt57
v3.0.36-rt58
v3.0.39-rt59

-rt56 was only the merge of 3.0.35. -rt 57 was only the merge of 3.0.36. -rt58 had only -rt specific changes applied. -rt59 was only the merge of 3.0.39.

Simply need to do a: git log v3.0.36-rt58 to see the -rt specific changes since -rt57 (the merge of 3.0.36).

Note, if we find a bug in mainline, we do not add it to the -rt series. Instead we forward a note to the mainline stable maintainer (Either Greg or Ben) and Cc the stable mailing list. The stable releases only get those fixes via mainline. Thus, you will be missing fixes that we find in mainline and push it back to them.

-- Steve

git log v3.0.36-rt58..3.0.48-rt72

That's what a source version control system is designed for AFAICT

Thanks,

tglx

Thanks for the tip. I (naively) presumed there were published changelogs and was looking for them. Nor did I know the git logs were limited to releases, and didn't look there because I feared a huge number of mid-release commits. Thanks for the pointer.

I've gone through all the commits from 3.0.36-rt58 to 3.0.48-rt72 and see a few items we need to pull in. The majority are irrelevant to our platform/configuration (ALSA, SCSI, x86, etc). But the few I did find point to us moving to rt72 when we can.

Thanks,
Pete

Ok. We've had 3.0.48-rt72 running on our setup that used to exhibit the corruption of the sequence count xt_recseq for over 24 hours. No further corruptions occurred. We've also tested this same version with our PPP setup and it has not exhibited any problems. Feel free to add a Tested-by tag on the 3.0.48-rt72 patch for the netfilter code.

Thanks,
Pete

On a separate note, I want to thank everyone that helped with this issue, especially Eric and Thomas, to and Steven and Thomas for schooling me on the changelog extraction. This problem was a big one for us that we were struggling to understand. All the help is greatly appreciated.

Thanks,
Pete

From: Steven Rostedt <[email protected]>

---
 localversion-rt |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index f38a3cc..70396d6 100644
@@ -1 +1 @@
--rt71
+-rt72-rc1

1.7.10.4