Skip to contentSkip to conversation

virtio/s390: fix race on airq_areas[]

PATCH In review Halil Pasic 3 participants 7 messages Jul 23, 2019 – Jul 24, 2019

Subsystems (5)
Stable branch KVM s390 virtio core +1 more subsystems
lore
Share & tools
lore

Halil Pasic

author maintainer
#0 lore raw

PATCH 1/1 virtio/s390: fix race on airq_areas[]

The access to airq_areas was racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit 39c7dcb15892 ("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit the airq_info that got overwritten was still functional. After that commit however the two infos share a summary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot with virtio_blk.

Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")

---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?

* I have a variant that does not need the extra mutex but uses cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.

---
 drivers/s390/virtio/virtio_ccw.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/s390/virtio/virtio_ccw.c +4 -0
index 1a55e5942d36..d97742662755 100644
@@ -145,6 +145,8 @@ struct airq_info {
 	struct airq_iv *aiv;
 };
 static struct airq_info *airq_areas[MAX_AIRQ_AREAS];
+DEFINE_MUTEX(airq_areas_lock);
+
 static u8 *summary_indicators;
 
 static inline u8 *get_summary_indicator(struct airq_info *info)
@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,
 	unsigned long bit, flags;
 
 	for (i = 0; i < MAX_AIRQ_AREAS && !indicator_addr; i++) {
+		mutex_lock(&airq_areas_lock);
 		if (!airq_areas[i])
 			airq_areas[i] = new_airq_info(i);
 		info = airq_areas[i];
+		mutex_unlock(&airq_areas_lock);
 		if (!info)
 			return 0;
 		write_lock_irqsave(&info->lock, flags);
Signature · 1 line
2.17.1
#1 lore raw
Quoted 8 lines · Halil Pasic

The access to airq_areas was racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit 39c7dcb15892 ("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit the airq_info that got overwritten was still functional. After that commit however the two infos share a summary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot with virtio_blk.

Quoted 6 lines
Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")

---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?

Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.

Quoted 4 lines

* I have a variant that does not need the extra mutex but uses cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.

This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.

Quoted 30 lines
---
 drivers/s390/virtio/virtio_ccw.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/s390/virtio/virtio_ccw.c +4 -0
index 1a55e5942d36..d97742662755 100644
@@ -145,6 +145,8 @@ struct airq_info {
 	struct airq_iv *aiv;
 };
 static struct airq_info *airq_areas[MAX_AIRQ_AREAS];
+DEFINE_MUTEX(airq_areas_lock);
+
 static u8 *summary_indicators;
 
 static inline u8 *get_summary_indicator(struct airq_info *info)
@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,
 	unsigned long bit, flags;
 
 	for (i = 0; i < MAX_AIRQ_AREAS && !indicator_addr; i++) {
+		mutex_lock(&airq_areas_lock);
 		if (!airq_areas[i])
 			airq_areas[i] = new_airq_info(i);
 		info = airq_areas[i];
+		mutex_unlock(&airq_areas_lock);
 		if (!info)
 			return 0;
 		write_lock_irqsave(&info->lock, flags);
maintainer Reviewed-by
#2 lore raw
Quoted 16 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic

The access to airq_areas was racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit 39c7dcb15892 ("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit the airq_info that got overwritten was still functional. After that commit however the two infos share a summary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot with virtio_blk.

Quoted 6 lines
Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")

---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?

Quoted 2 lines

Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.

Yes, this needs to be cc:stable.

Quoted 8 lines
Quoted 5 lines

* I have a variant that does not need the extra mutex but uses cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.

This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.

Yes, this is ultimately called through the ->probe functions of virtio drivers.

Quoted 20 lines
Quoted 20 lines
---
 drivers/s390/virtio/virtio_ccw.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/s390/virtio/virtio_ccw.c +2 -0
index 1a55e5942d36..d97742662755 100644
@@ -145,6 +145,8 @@ struct airq_info {
 	struct airq_iv *aiv;
 };
 static struct airq_info *airq_areas[MAX_AIRQ_AREAS];
+DEFINE_MUTEX(airq_areas_lock);
+
 static u8 *summary_indicators;
 
 static inline u8 *get_summary_indicator(struct airq_info *info)
@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,
 	unsigned long bit, flags;
 
Quoted 11 lines

drivers/s390/virtio/virtio_ccw.c static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs, · line 277

Quoted 10 lines
drivers/s390/virtio/virtio_ccw.c +2 -0
 	for (i = 0; i < MAX_AIRQ_AREAS && !indicator_addr; i++) {
+		mutex_lock(&airq_areas_lock);
 		if (!airq_areas[i])
 			airq_areas[i] = new_airq_info(i);
 		info = airq_areas[i];
+		mutex_unlock(&airq_areas_lock);
 		if (!info)
 			return 0;
 		write_lock_irqsave(&info->lock, flags);

Should I pick this and send a pull request, or is it quicker to just take this directly?

#3 lore raw
Quoted 68 lines · Cornelia Huck
Quoted 16 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic

The access to airq_areas was racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit 39c7dcb15892 ("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit the airq_info that got overwritten was still functional. After that commit however the two infos share a summary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot with virtio_blk.

Quoted 6 lines
Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")

---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?

Quoted 2 lines

Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.

Yes, this needs to be cc:stable.

Quoted 8 lines
Quoted 5 lines

* I have a variant that does not need the extra mutex but uses cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.

This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.

Yes, this is ultimately called through the ->probe functions of virtio drivers.

Quoted 20 lines
Quoted 20 lines
---
 drivers/s390/virtio/virtio_ccw.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/s390/virtio/virtio_ccw.c +2 -0
index 1a55e5942d36..d97742662755 100644
@@ -145,6 +145,8 @@ struct airq_info {
 	struct airq_iv *aiv;
 };
 static struct airq_info *airq_areas[MAX_AIRQ_AREAS];
+DEFINE_MUTEX(airq_areas_lock);
+
 static u8 *summary_indicators;
 
 static inline u8 *get_summary_indicator(struct airq_info *info)
@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,
 	unsigned long bit, flags;
 
Quoted 11 lines

drivers/s390/virtio/virtio_ccw.c static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs, · line 277

Quoted 10 lines
drivers/s390/virtio/virtio_ccw.c +2 -0
 	for (i = 0; i < MAX_AIRQ_AREAS && !indicator_addr; i++) {
+		mutex_lock(&airq_areas_lock);
 		if (!airq_areas[i])
 			airq_areas[i] = new_airq_info(i);
 		info = airq_areas[i];
+		mutex_unlock(&airq_areas_lock);
 		if (!info)
 			return 0;
 		write_lock_irqsave(&info->lock, flags);
Quoted 4 lines

Should I pick this and send a pull request, or is it quicker to just take this directly?

I think we can you did via a fast path. Halil, can you push to the s390 tree?

author maintainer
#4 lore raw
Quoted 71 lines · On Wed, 24 Jul 2019 10:39:13 +0200 Christian Borntraeger…
Quoted 68 lines · Cornelia Huck
Quoted 16 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic

The access to airq_areas was racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit 39c7dcb15892 ("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit the airq_info that got overwritten was still functional. After that commit however the two infos share a summary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot with virtio_blk.

Quoted 6 lines
Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")

---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?

Quoted 2 lines

Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.

Yes, this needs to be cc:stable.

Quoted 8 lines
Quoted 5 lines

* I have a variant that does not need the extra mutex but uses cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.

This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.

Yes, this is ultimately called through the ->probe functions of virtio drivers.

Quoted 31 lines
Quoted 30 lines
---
 drivers/s390/virtio/virtio_ccw.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/s390/virtio/virtio_ccw.c +4 -0
index 1a55e5942d36..d97742662755 100644
@@ -145,6 +145,8 @@ struct airq_info {
 	struct airq_iv *aiv;
 };
 static struct airq_info *airq_areas[MAX_AIRQ_AREAS];
+DEFINE_MUTEX(airq_areas_lock);
+
 static u8 *summary_indicators;
 
 static inline u8 *get_summary_indicator(struct airq_info *info)
@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,
 	unsigned long bit, flags;
 
 	for (i = 0; i < MAX_AIRQ_AREAS && !indicator_addr; i++) {
+		mutex_lock(&airq_areas_lock);
 		if (!airq_areas[i])
 			airq_areas[i] = new_airq_info(i);
 		info = airq_areas[i];
+		mutex_unlock(&airq_areas_lock);
 		if (!info)
 			return 0;
 		write_lock_irqsave(&info->lock, flags);
Quoted 6 lines
Quoted 4 lines

Should I pick this and send a pull request, or is it quicker to just take this directly?

I think we can you did via a fast path. Halil, can you push to the s390 tree?

Sure!

Regards,
Halil

Quoted 1 line
author maintainer
#5 lore raw
In reply to Christian Borntraeger
Quoted 18 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic

The access to airq_areas was racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit 39c7dcb15892 ("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit the airq_info that got overwritten was still functional. After that commit however the two infos share a summary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot with virtio_blk.

Quoted 6 lines
Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")

---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?

Quoted 2 lines

Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.

No I would not like to make an attempt at proving that. I prefer code race free anyway. CC-ing stable.

Quoted 8 lines
Quoted 5 lines

* I have a variant that does not need the extra mutex but uses cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.

This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.

Right, this is only relevant during device initialization, which is an infrequent operation.

Thanks,
Halil

Quoted 31 lines
Quoted 30 lines
---
 drivers/s390/virtio/virtio_ccw.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/s390/virtio/virtio_ccw.c +4 -0
index 1a55e5942d36..d97742662755 100644
@@ -145,6 +145,8 @@ struct airq_info {
 	struct airq_iv *aiv;
 };
 static struct airq_info *airq_areas[MAX_AIRQ_AREAS];
+DEFINE_MUTEX(airq_areas_lock);
+
 static u8 *summary_indicators;
 
 static inline u8 *get_summary_indicator(struct airq_info *info)
@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,
 	unsigned long bit, flags;
 
 	for (i = 0; i < MAX_AIRQ_AREAS && !indicator_addr; i++) {
+		mutex_lock(&airq_areas_lock);
 		if (!airq_areas[i])
 			airq_areas[i] = new_airq_info(i);
 		info = airq_areas[i];
+		mutex_unlock(&airq_areas_lock);
 		if (!info)
 			return 0;
 		write_lock_irqsave(&info->lock, flags);
author maintainer
#6 lore raw
In reply to Christian Borntraeger
Quoted 18 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic

The access to airq_areas was racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit 39c7dcb15892 ("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit the airq_info that got overwritten was still functional. After that commit however the two infos share a summary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot with virtio_blk.

Quoted 6 lines
Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")

---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?

Quoted 2 lines

Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.

No I would not like to make an attempt at proving that. I prefer code race free anyway. CC-ing stable.

Quoted 8 lines
Quoted 5 lines

* I have a variant that does not need the extra mutex but uses cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.

This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.

Right, this is only relevant during device initialization, which is an infrequent operation.

Thanks,
Halil

Quoted 31 lines
Quoted 30 lines
---
 drivers/s390/virtio/virtio_ccw.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/s390/virtio/virtio_ccw.c +4 -0
index 1a55e5942d36..d97742662755 100644
@@ -145,6 +145,8 @@ struct airq_info {
 	struct airq_iv *aiv;
 };
 static struct airq_info *airq_areas[MAX_AIRQ_AREAS];
+DEFINE_MUTEX(airq_areas_lock);
+
 static u8 *summary_indicators;
 
 static inline u8 *get_summary_indicator(struct airq_info *info)
@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,
 	unsigned long bit, flags;
 
 	for (i = 0; i < MAX_AIRQ_AREAS && !indicator_addr; i++) {
+		mutex_lock(&airq_areas_lock);
 		if (!airq_areas[i])
 			airq_areas[i] = new_airq_info(i);
 		info = airq_areas[i];
+		mutex_unlock(&airq_areas_lock);
 		if (!info)
 			return 0;
 		write_lock_irqsave(&info->lock, flags);