virtio/s390: fix race on airq_areas[]
Subsystems (5)
Halil Pasic
maintainerPATCH 1/1 virtio/s390: fix race on airq_areas[]
The access to airq_areas was racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit 39c7dcb15892 ("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit the airq_info that got overwritten was still functional. After that commit however the two infos share a summary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot with virtio_blk.
---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?
* I have a variant that does not need the extra mutex but uses cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.
--- drivers/s390/virtio/virtio_ccw.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/s390/virtio/virtio_ccw.c +4 -0
@@ -145,6 +145,8 @@ struct airq_info {struct airq_iv *aiv; };static struct airq_info *airq_areas[MAX_AIRQ_AREAS ]; +DEFINE_MUTEX (airq_areas_lock); +static u8 *summary_indicators;static inline u8 *get_summary_indicator (struct airq_info *info)
@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,unsigned long bit, flags;for (i =0 ; i <MAX_AIRQ_AREAS && !indicator_addr; i++) { +mutex_lock (&airq_areas_lock);if (!airq_areas[i]) airq_areas[i] =new_airq_info (i); info = airq_areas[i]; +mutex_unlock (&airq_areas_lock);if (!info)return 0 ;write_lock_irqsave (&info->lock, flags);
Signature · 1 line
2.17.1
Quoted 8 lines · Halil Pasic
The access to
airq_areaswas racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit39c7dcb15892("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit theairq_infothat got overwritten was still functional. After that commit however the two infos share asummary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot withvirtio_blk.
Quoted 6 lines
Signed-off-by Halil Pasic <[email protected]>Reported-by Marc Hartmayer <[email protected]>Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?
Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.
Quoted 4 lines
* I have a variant that does not need the extra mutex but uses
cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.
This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.
Quoted 30 lines
--- drivers/s390/virtio/virtio_ccw.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/s390/virtio/virtio_ccw.c +4
-0@@ -145,6 +145,8 @@ struct airq_info {struct airq_iv *aiv; };static struct airq_info *airq_areas[MAX_AIRQ_AREAS ]; +DEFINE_MUTEX (airq_areas_lock); +static u8 *summary_indicators;static inline u8 *get_summary_indicator (struct airq_info *info)@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,unsigned long bit, flags;for (i =0 ; i <MAX_AIRQ_AREAS && !indicator_addr; i++) { +mutex_lock (&airq_areas_lock);if (!airq_areas[i]) airq_areas[i] =new_airq_info (i); info = airq_areas[i]; +mutex_unlock (&airq_areas_lock);if (!info)return 0 ;write_lock_irqsave (&info->lock, flags);
Quoted 16 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic
The access to
airq_areaswas racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit39c7dcb15892("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit theairq_infothat got overwritten was still functional. After that commit however the two infos share asummary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot withvirtio_blk.Quoted 6 lines
Signed-off-by Halil Pasic <[email protected]>Reported-by Marc Hartmayer <[email protected]>Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?
Quoted 2 lines
Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.
Yes, this needs to be cc:stable.
Quoted 8 lines
Quoted 5 lines
* I have a variant that does not need the extra mutex but uses
cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.
Yes, this is ultimately called through the ->probe functions of virtio drivers.
Quoted 20 lines
Quoted 20 lines
--- drivers/s390/virtio/virtio_ccw.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/s390/virtio/virtio_ccw.c +2
-0@@ -145,6 +145,8 @@ struct airq_info {struct airq_iv *aiv; };static struct airq_info *airq_areas[MAX_AIRQ_AREAS ]; +DEFINE_MUTEX (airq_areas_lock); +static u8 *summary_indicators;static inline u8 *get_summary_indicator (struct airq_info *info)@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,unsigned long bit, flags;
Quoted 11 lines
drivers/s390/virtio/virtio_ccw.c static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs, · line 277
Quoted 10 lines
drivers/s390/virtio/virtio_ccw.c +2
-0for (i =0 ; i <MAX_AIRQ_AREAS && !indicator_addr; i++) { +mutex_lock (&airq_areas_lock);if (!airq_areas[i]) airq_areas[i] =new_airq_info (i); info = airq_areas[i]; +mutex_unlock (&airq_areas_lock);if (!info)return 0 ;write_lock_irqsave (&info->lock, flags);
Should I pick this and send a pull request, or is it quicker to just take this directly?
Quoted 68 lines · Cornelia Huck
Quoted 16 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic
The access to
airq_areaswas racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit39c7dcb15892("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit theairq_infothat got overwritten was still functional. After that commit however the two infos share asummary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot withvirtio_blk.Quoted 6 lines
Signed-off-by Halil Pasic <[email protected]>Reported-by Marc Hartmayer <[email protected]>Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?Quoted 2 lines
Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.
Yes, this needs to be cc:stable.
Quoted 8 lines
Quoted 5 lines
* I have a variant that does not need the extra mutex but uses
cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.
Yes, this is ultimately called through the ->probe functions of virtio drivers.
Quoted 20 lines
Quoted 20 lines
--- drivers/s390/virtio/virtio_ccw.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/s390/virtio/virtio_ccw.c +2
-0@@ -145,6 +145,8 @@ struct airq_info {struct airq_iv *aiv; };static struct airq_info *airq_areas[MAX_AIRQ_AREAS ]; +DEFINE_MUTEX (airq_areas_lock); +static u8 *summary_indicators;static inline u8 *get_summary_indicator (struct airq_info *info)@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,unsigned long bit, flags;Quoted 11 lines
drivers/s390/virtio/virtio_ccw.c static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs, · line 277
Quoted 10 lines
drivers/s390/virtio/virtio_ccw.c +2
-0for (i =0 ; i <MAX_AIRQ_AREAS && !indicator_addr; i++) { +mutex_lock (&airq_areas_lock);if (!airq_areas[i]) airq_areas[i] =new_airq_info (i); info = airq_areas[i]; +mutex_unlock (&airq_areas_lock);if (!info)return 0 ;write_lock_irqsave (&info->lock, flags);
Quoted 4 lines
Reviewed-by Cornelia Huck <[email protected]>Should I pick this and send a pull request, or is it quicker to just take this directly?
I think we can you did via a fast path. Halil, can you push to the s390 tree?
Halil Pasic
maintainerQuoted 71 lines · On Wed, 24 Jul 2019 10:39:13 +0200 Christian Borntraeger…
Quoted 68 lines · Cornelia Huck
Quoted 16 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic
The access to
airq_areaswas racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit39c7dcb15892("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit theairq_infothat got overwritten was still functional. After that commit however the two infos share asummary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot withvirtio_blk.Quoted 6 lines
Signed-off-by Halil Pasic <[email protected]>Reported-by Marc Hartmayer <[email protected]>Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?Quoted 2 lines
Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.
Yes, this needs to be cc:stable.
Quoted 8 lines
Quoted 5 lines
* I have a variant that does not need the extra mutex but uses
cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.
Yes, this is ultimately called through the ->probe functions of virtio drivers.
Quoted 31 lines
Quoted 30 lines
--- drivers/s390/virtio/virtio_ccw.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/s390/virtio/virtio_ccw.c +4
-0@@ -145,6 +145,8 @@ struct airq_info {struct airq_iv *aiv; };static struct airq_info *airq_areas[MAX_AIRQ_AREAS ]; +DEFINE_MUTEX (airq_areas_lock); +static u8 *summary_indicators;static inline u8 *get_summary_indicator (struct airq_info *info)@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,unsigned long bit, flags;for (i =0 ; i <MAX_AIRQ_AREAS && !indicator_addr; i++) { +mutex_lock (&airq_areas_lock);if (!airq_areas[i]) airq_areas[i] =new_airq_info (i); info = airq_areas[i]; +mutex_unlock (&airq_areas_lock);if (!info)return 0 ;write_lock_irqsave (&info->lock, flags);
Quoted 6 lines
Quoted 4 lines
Reviewed-by Cornelia Huck <[email protected]>Should I pick this and send a pull request, or is it quicker to just take this directly?
I think we can you did via a fast path. Halil, can you push to the s390 tree?
Sure!
Regards,
Halil
Quoted 1 line
Halil Pasic
maintainerQuoted 18 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic
The access to
airq_areaswas racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit39c7dcb15892("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit theairq_infothat got overwritten was still functional. After that commit however the two infos share asummary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot withvirtio_blk.Quoted 6 lines
Signed-off-by Halil Pasic <[email protected]>Reported-by Marc Hartmayer <[email protected]>Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?
Quoted 2 lines
Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.
No I would not like to make an attempt at proving that. I prefer code race free anyway. CC-ing stable.
Quoted 8 lines
Quoted 5 lines
* I have a variant that does not need the extra mutex but uses
cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.
Right, this is only relevant during device initialization, which is an infrequent operation.
Thanks,
Halil
Quoted 31 lines
Quoted 30 lines
--- drivers/s390/virtio/virtio_ccw.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/s390/virtio/virtio_ccw.c +4
-0@@ -145,6 +145,8 @@ struct airq_info {struct airq_iv *aiv; };static struct airq_info *airq_areas[MAX_AIRQ_AREAS ]; +DEFINE_MUTEX (airq_areas_lock); +static u8 *summary_indicators;static inline u8 *get_summary_indicator (struct airq_info *info)@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,unsigned long bit, flags;for (i =0 ; i <MAX_AIRQ_AREAS && !indicator_addr; i++) { +mutex_lock (&airq_areas_lock);if (!airq_areas[i]) airq_areas[i] =new_airq_info (i); info = airq_areas[i]; +mutex_unlock (&airq_areas_lock);if (!info)return 0 ;write_lock_irqsave (&info->lock, flags);
Halil Pasic
maintainerQuoted 18 lines · On Wed, 24 Jul 2019 08:44:19 +0200 Christian Borntraeger…
Quoted 8 lines · Halil Pasic
The access to
airq_areaswas racy ever since the adapter interrupts got introduced to virtio-ccw, but since commit39c7dcb15892("virtio/s390:
make airq summary indicators DMA") this became an issue in practice as well. Namely before that commit theairq_infothat got overwritten was still functional. After that commit however the two infos share asummary_indicator, which aggravates the situation. Which means auto-online mechanism occasionally hangs the boot withvirtio_blk.Quoted 6 lines
Signed-off-by Halil Pasic <[email protected]>Reported-by Marc Hartmayer <[email protected]>Fixes 96b14536d935 ("virtio-ccw: virtio-ccw adapter interrupt support.")---
* We need definitely this fixed for 5.3. For older stable kernels it is to be discussed. @Connie what do you think: do we need a cc stable?
Quoted 2 lines
Unless you can prove that the problem could never happen on old version we absolutely do need cc stable.
No I would not like to make an attempt at proving that. I prefer code race free anyway. CC-ing stable.
Quoted 8 lines
Quoted 5 lines
* I have a variant that does not need the extra mutex but uses
cmpxchg(). Decided to post this one because that one is more complex. But if there is interest we can have a look at it as well.This is slow path (startup) and never called in hot path. Correct? Mutex should be fine.
Right, this is only relevant during device initialization, which is an infrequent operation.
Thanks,
Halil
Quoted 31 lines
Quoted 30 lines
--- drivers/s390/virtio/virtio_ccw.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/s390/virtio/virtio_ccw.c +4
-0@@ -145,6 +145,8 @@ struct airq_info {struct airq_iv *aiv; };static struct airq_info *airq_areas[MAX_AIRQ_AREAS ]; +DEFINE_MUTEX (airq_areas_lock); +static u8 *summary_indicators;static inline u8 *get_summary_indicator (struct airq_info *info)@@ -265,9 +267,11 @@ static unsigned long get_airq_indicator(struct virtqueue *vqs[], int nvqs,unsigned long bit, flags;for (i =0 ; i <MAX_AIRQ_AREAS && !indicator_addr; i++) { +mutex_lock (&airq_areas_lock);if (!airq_areas[i]) airq_areas[i] =new_airq_info (i); info = airq_areas[i]; +mutex_unlock (&airq_areas_lock);if (!info)return 0 ;write_lock_irqsave (&info->lock, flags);