Skip to contentSkip to conversation

Report in downstream Debian: mpt3sas broken with xen dom0 with update to 5.10.149 in 5.10.y.

DISCUSSION Salvatore Bonaccorso 9 participants 19 messages Oct 21, 2022 – Mar 14, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

We got the following report in Debian after an update from 5.10.140 to the current 5.10.149. Full quoting below (from
https://bugs.debian.org/1022126). Does this ring some bell about known regressions?

Quoted 27 lines · Adi Kriegisch

Package: linux-image-5.10.0-19-amd64
Version: 5.10.149-1
Severity: important

Dear maintainers,

with the upgrade to the latest bullseye kernel (5.10.149-1), our xen setup is unbootable due to swiotlb buffer errors:
| sd 0:0:0:0: scsi_dma_map failed: request for 401408 bytes!
and
| mpt3sas 0000:01:00.0: swiotlb buffer is full (sz: 401408 bytes),
| total 32768 (slots), used 0 (slots)
(the byte sizes vary between boots).

After reading bug #850425[1], we also tried to force 32bit mode in the mpt3sas driver by specifying a dom0 memory below 4G; this lets the machine boot, but almost immediately after that fails with the same error. Notable difference is that the used slots are 128.

Xen commandline:
dom0_mem=4096M,max:4096M dom0_max_vcpus=4 dom0_vcpus_pin ucode=scan xpti=dom0=false,domu=true gnttab_max_frames=128

Using dom0-iommu=map-inclusive in some combinations with swiotlb on the kernel commandline gives us some used slots (way below 128) in the error message even in 64bit dma mode in the mpt3sas driver.

Quoted 8 lines

The kernel works when booted without xen. We'd be more than happy to get pointers on how to fix that issue or patches to test!

Thanks for your help!

-- Adi

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850425

Regards,
Salvatore

#1 lore raw
Quoted 4 lines · Salvatore Bonaccorso

We got the following report in Debian after an update from 5.10.140 to the current 5.10.149. Full quoting below (from
https://bugs.debian.org/1022126). Does this ring some bell about known regressions?

Only three mpt3sas changes are new in v5.10.149 compared to v5.10.140:

$ git log --format=oneline v5.10.140..v5.10.149
2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()
e7fafef9830c4a01e60f76e3860a9bef0262378d scsi: mpt3sas: Force PCIe scatterlist allocations to be within same 4 GB region
ea10a652ad2ae2cf3eced6f632a5c98f26727057 scsi: mpt3sas: Fix use-after-free warning

Sreekanth and Suganath, can you help with bisecting this issue? For the full report, see also https://lore.kernel.org/linux-scsi/[email protected]/.

Thanks,

Bart.

maintainer
#2 lore raw
Quoted 7 lines · Bart Van Assche
Quoted 4 lines · Salvatore Bonaccorso

We got the following report in Debian after an update from 5.10.140 to the current 5.10.149. Full quoting below (from
https://bugs.debian.org/1022126). Does this ring some bell about known regressions?

Quoted 8 lines

Only three mpt3sas changes are new in v5.10.149 compared to v5.10.140:

$ git log --format=oneline v5.10.140..v5.10.149
2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()
e7fafef9830c4a01e60f76e3860a9bef0262378d scsi: mpt3sas: Force PCIe scatterlist allocations to be within same 4 GB region
ea10a652ad2ae2cf3eced6f632a5c98f26727057 scsi: mpt3sas: Fix use-after-free warning

Sreekanth and Suganath, can you help with bisecting this issue? For the full report, see also https://lore.kernel.org/linux-scsi/[email protected]/.

This issue is getting observed after having the below patch changes, 2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()

What is happening is that on Xen hypervisor, this dma_get_required_mask() API always returns a 32 bit DMA mask. I.e. It says that the minimum DMA mask required to access the host memory is 32 bit and hence mpt3sas driver is setting the DMA mask to 32bit. So, on a 64 bit machine, if the driver set's the DMA mask to 32 bit then SWIOTLB's bounce buffer comes into picture during IOs. Since these bounce buffers are limited in size and hence we observe the IO hang if the large IOs are issued.

I am not sure whether this API's return value is correct or not in the Xen environment. If it is correct then I have to modify the driver to not use this API and directly set the DMA mask to 64 bit if the system is a 64bit machine.

Thanks,
Sreekanth

Quoted 4 lines

Thanks,

Bart.

#3 lore raw
Quoted 30 lines · Sreekanth Reddy
Quoted 7 lines · Bart Van Assche
Quoted 4 lines · Salvatore Bonaccorso

We got the following report in Debian after an update from 5.10.140 to the current 5.10.149. Full quoting below (from
https://bugs.debian.org/1022126). Does this ring some bell about known regressions?

Quoted 8 lines

Only three mpt3sas changes are new in v5.10.149 compared to v5.10.140:

$ git log --format=oneline v5.10.140..v5.10.149
2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()
e7fafef9830c4a01e60f76e3860a9bef0262378d scsi: mpt3sas: Force PCIe scatterlist allocations to be within same 4 GB region
ea10a652ad2ae2cf3eced6f632a5c98f26727057 scsi: mpt3sas: Fix use-after-free warning

Sreekanth and Suganath, can you help with bisecting this issue? For the full report, see also https://lore.kernel.org/linux-scsi/[email protected]/.

This issue is getting observed after having the below patch changes, 2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()

What is happening is that on Xen hypervisor, this dma_get_required_mask() API always returns a 32 bit DMA mask. I.e. It says that the minimum DMA mask required to access the host memory is 32 bit and hence mpt3sas driver is setting the DMA mask to 32bit. So, on a 64 bit machine, if the driver set's the DMA mask to 32 bit then SWIOTLB's bounce buffer comes into picture during IOs. Since these bounce buffers are limited in size and hence we observe the IO hang if the large IOs are issued.

Quoted 4 lines

I am not sure whether this API's return value is correct or not in the Xen environment. If it is correct then I have to modify the driver to not use this API and directly set the DMA mask to 64 bit if the system is a 64bit machine.

Please recheck the backported patch in 5.10.y. It is _wrong_. The backport has:

drivers/scsi/mpt3sas/mpt3sas_base.c +1 -1
@@ -2993,7 +2993,7 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 
        if (ioc->is_mcpu_endpoint ||
            sizeof(dma_addr_t) == 4 || ioc->use_32bit_dma ||
-           dma_get_required_mask(&pdev->dev) <= 32)
+           dma_get_required_mask(&pdev->dev) <= DMA_BIT_MASK(32))
                ioc->dma_mask = 32;
        /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
        else if (ioc->hba_mpi_version_belonged > MPI2_VERSION)

While the upstream patch has:

+ if (ioc->is_mcpu_endpoint || sizeof(dma_addr_t) == 4 ||

+           dma_get_required_mask(&pdev->dev) <= 32) {
                ioc->dma_mask = 32;

+ coherent_dma_mask = dma_mask = DMA_BIT_MASK(32);

Juergen

#7 lore raw
In reply to Sreekanth Reddy
Quoted 25 lines · Sreekanth Reddy
Quoted 18 lines · Bart Van Assche
Quoted 6 lines · Salvatore Bonaccorso

We got the following report in Debian after an update from 5.10.140 to
the current 5.10.149. Full quoting below (from
https://bugs.debian.org/1022126). Does this ring some bell about known
regressions?

Only three mpt3sas changes are new in v5.10.149 compared to v5.10.140:

$ git log --format=oneline v5.10.140..v5.10.149
2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return
value check of dma_get_required_mask()
e7fafef9830c4a01e60f76e3860a9bef0262378d scsi: mpt3sas: Force PCIe
scatterlist allocations to be within same 4 GB region
ea10a652ad2ae2cf3eced6f632a5c98f26727057 scsi: mpt3sas: Fix use-
after-free warning
Quoted 4 lines

Sreekanth and Suganath, can you help with bisecting this issue? For the
full report, see also
https://lore.kernel.org/linux-scsi/[email protected]/.

Quoted 8 lines

This issue is getting observed after having the below patch changes, 2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()

What is happening is that on Xen hypervisor, this dma_get_required_mask() API always returns a 32 bit DMA mask. I.e. It says that the minimum DMA mask required to access the host memory is 32 bit and hence mpt3sas driver is setting the DMA mask to 32bit.

This sounds entirely correct because the VM is booted with (from the original debian bug report):

dom0_mem=4096M,max:4096M dom0_max_vcpus=4 dom0_vcpus_pin
ucode=scan xpti=dom0=false,domu=true gnttab_max_frames=128

So it has no memory above 4GB and thus 32 bit addressing is the minimum required. If you boot a machine with >4GB and Xen still returns a 32 bit mask here, then we have a Xen problem.

Quoted 4 lines

So, on a 64 bit machine, if the driver set's the DMA mask to 32 bit
then SWIOTLB's bounce buffer comes into picture during IOs. Since these bounce buffers are limited in size and hence we observe the IO hang if the large IOs are issued.

Why is the SWIOTLB active if all the physical memory in the VM is within the range of the DMA mask? If this is really happening, it sounds like a SWIOTLB bug.

Quoted 4 lines

I am not sure whether this API's return value is correct or not in the Xen environment. If it is correct then I have to modify the driver to not use this API and directly set the DMA mask to 64 bit if the system is a 64bit machine.

The original design of the API is to describe exactly the minimum direct DMA requirements. There are a large number of cards with multiple DMA register formats, the most common being to use either a compact 32 bit or an expanded 64 bit register to describe a page location. The former gives 39 bits of addressing and the latter 64. If the DMA mask is 39 bits or below as described by this API, then the card can use the compact address form.

James

#8 lore raw
Quoted 38 lines · James Bottomley
Quoted 25 lines · Sreekanth Reddy
Quoted 18 lines · Bart Van Assche
Quoted 6 lines · Salvatore Bonaccorso

We got the following report in Debian after an update from 5.10.140 to
the current 5.10.149. Full quoting below (from
https://bugs.debian.org/1022126). Does this ring some bell about known
regressions?

Only three mpt3sas changes are new in v5.10.149 compared to v5.10.140:

$ git log --format=oneline v5.10.140..v5.10.149
2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return
value check of dma_get_required_mask()
e7fafef9830c4a01e60f76e3860a9bef0262378d scsi: mpt3sas: Force PCIe
scatterlist allocations to be within same 4 GB region
ea10a652ad2ae2cf3eced6f632a5c98f26727057 scsi: mpt3sas: Fix use-
after-free warning
Quoted 4 lines

Sreekanth and Suganath, can you help with bisecting this issue? For the
full report, see also
https://lore.kernel.org/linux-scsi/[email protected]/.

Quoted 8 lines

This issue is getting observed after having the below patch changes, 2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()

What is happening is that on Xen hypervisor, this dma_get_required_mask() API always returns a 32 bit DMA mask. I.e. It says that the minimum DMA mask required to access the host memory is 32 bit and hence mpt3sas driver is setting the DMA mask to 32bit.

This sounds entirely correct because the VM is booted with (from the original debian bug report):

Quoted 6 lines

dom0_mem=4096M,max:4096M dom0_max_vcpus=4 dom0_vcpus_pin
ucode=scan xpti=dom0=false,domu=true gnttab_max_frames=128

So it has no memory above 4GB and thus 32 bit addressing is the minimum required. If you boot a machine with >4GB and Xen still returns a 32 bit mask here, then we have a Xen problem.

Dom0 is (normally) a PV domain, so the physical memory can be still above 4 GB even with dom0_mem set to 4GB.

But please see my other reply to this thread. I think at least one of the backports is plain wrong.

Juergen

#17 lore raw
In reply to Sreekanth Reddy

Hi,

Quoted 14 lines · Sreekanth Reddy
Quoted 13 lines · Bart Van Assche
Quoted 4 lines · Salvatore Bonaccorso

We got the following report in Debian after an update from 5.10.140 to the current 5.10.149. Full quoting below (from
https://bugs.debian.org/1022126). Does this ring some bell about known regressions?

Only three mpt3sas changes are new in v5.10.149 compared to v5.10.140:

$ git log --format=oneline v5.10.140..v5.10.149
2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()
e7fafef9830c4a01e60f76e3860a9bef0262378d scsi: mpt3sas: Force PCIe scatterlist allocations to be within same 4 GB region
ea10a652ad2ae2cf3eced6f632a5c98f26727057 scsi: mpt3sas: Fix use-after-free warning
Quoted 6 lines
Quoted 2 lines

Sreekanth and Suganath, can you help with bisecting this issue? For the full report, see also https://lore.kernel.org/linux-scsi/[email protected]/.

This issue is getting observed after having the below patch changes, 2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()

reverting this patch fixed the issue for us.

Quoted 8 lines

What is happening is that on Xen hypervisor, this dma_get_required_mask() API always returns a 32 bit DMA mask. I.e. It says that the minimum DMA mask required to access the host memory is 32 bit and hence mpt3sas driver is setting the DMA mask to 32bit. So, on a 64 bit machine, if the driver set's the DMA mask to 32 bit then SWIOTLB's bounce buffer comes into picture during IOs. Since these bounce buffers are limited in size and hence we observe the IO hang if the large IOs are issued.

I am not sure about what is going on here but while reverting the above patch, I added a printk right above the if:
| printk("dma_get_required_mask =%lld\n",dma_get_required_mask(&pdev->dev));
which always (as in booting dom0 with 2GB, 4GB, 16GB and booting bare metal with 256GB) returns 4294967295 (DMA_BIT_MASK(32)).

Quoted 4 lines

I am not sure whether this API's return value is correct or not in the Xen environment. If it is correct then I have to modify the driver to not use this API and directly set the DMA mask to 64 bit if the system is a 64bit machine.

Obviously, our server always reports a 32bit mask which works just fine in the bare metal case but does not in the xen dom0 case. Is there anything I can do to help tracking the issue down?

thanks and all the best,
Adi

Quoted 9 lines

Thanks,
Sreekanth

Quoted 4 lines

Thanks,

Bart.

#9 lore raw
In reply to Juergen Gross
Quoted 2 lines · Juergen Gross

Dom0 is (normally) a PV domain, so the physical memory can be still above 4 GB even with dom0_mem set to 4GB.

Which means that we need to ensure the DMA ops for Xen-PV (which is always xen-swiotlb I think?) need to return DMA_BIT_MASK(64) or whatever is the highest possible address.

#10 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Juergen Gross

Dom0 is (normally) a PV domain, so the physical memory can be still above 4 GB even with dom0_mem set to 4GB.

Which means that we need to ensure the DMA ops for Xen-PV (which is always xen-swiotlb I think?) need to return DMA_BIT_MASK(64) or whatever is the highest possible address.

Xen tries to allocate memory from the top down, in order to keep the low memory free for special purses.  (32bit PV guests strictly need to live below the 128G boundary in host physical space for ABI reasons, and Xen's struct vcpu needs to be below the 4G boundary for compatibility with some processors.)

This means that a 64bit PV dom0 ends up occupying the highest addresses in RAM.  Naturally, this is a problem for devices which can't do 64bit DMA.

The xen-swiotlb is a region (default 128M iirc) which dom0 requests of Xen during startup.  It is below the 4G boundary, and physically contiguous (unlike most of the rest of PV guest memory).

Hence it's a suitable bounce buffer for any 32-bit-only devices.

I don't know exactly how this translates to Linux internals, but most devices are fine and it's routinely the mpt2/3sas drivers which encounter problems.  It would be lovely if we could get to the bottom of this for once and for all.

~Andrew

#18 lore raw
In reply to Sreekanth Reddy
Quoted 3 lines · Sreekanth Reddy

This issue is getting observed after having the below patch changes, 2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()

Looking at this commit it seems odd. dma_get_required_mask() should only be used as an optimization for hardware that actually benefits from a lower DMA Mask. That means either classic PCI that requires DAC cycles, or firmware architectures like aic7xxx that do need additional overhead. I don't think either is the case for mpt3sas, so I think (in addition to fixing up the Xen required mask), mpt3sas should do something like:

drivers/scsi/mpt3sas/mpt3sas_base.c +1 -2
index 4e981ccaac4163..295942a8989780 100644
@@ -2992,8 +2992,7 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 	struct sysinfo s;
 	u64 coherent_dma_mask, dma_mask;
 
-	if (ioc->is_mcpu_endpoint || sizeof(dma_addr_t) == 4 ||
-	    dma_get_required_mask(&pdev->dev) <= DMA_BIT_MASK(32)) {
+	if (ioc->is_mcpu_endpoint) {
 		ioc->dma_mask = 32;
 		coherent_dma_mask = dma_mask = DMA_BIT_MASK(32);
 	/* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
#11 lore raw
In reply to Andrew Cooper
Quoted 4 lines · Andrew Cooper

I don't know exactly how this translates to Linux internals, but most devices are fine and it's routinely the mpt2/3sas drivers which encounter problems.  It would be lovely if we could get to the bottom of this for once and for all.

So to summarize my two mails: I think te use of dma_get_required_mask in mpt3sas is wrong, and the dma_get_required_mask return value from xen-swiotlb is also wrong. Fixing either one should fix this problem, and I think we should fix both.

#4 lore raw
In reply to Juergen Gross

On Mon, 2022-10-24 at 14:55 +0200, Juergen Gross wrote:
[...]

Quoted 21 lines

Please recheck the backported patch in 5.10.y. It is _wrong_. The backport
has:

drivers/scsi/mpt3sas/mpt3sas_base.c +1 -1
@@ -2993,7 +2993,7 @@ _base_config_dma_addressing(struct
 MPT3SAS_ADAPTER *ioc, 
 struct pci_dev *pdev)
 
         if (ioc->is_mcpu_endpoint ||
             sizeof(dma_addr_t) == 4 || ioc->use_32bit_dma ||
-           dma_get_required_mask(&pdev->dev) <= 32)
+           dma_get_required_mask(&pdev->dev) <= DMA_BIT_MASK(32))
                 ioc->dma_mask = 32;
         /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
         else if (ioc->hba_mpi_version_belonged > MPI2_VERSION)

While the upstream patch has:

drivers/scsi/mpt3sas/mpt3sas_base.c

Quoted 4 lines
drivers/scsi/mpt3sas/mpt3sas_base.c +3 -0
+       if (ioc->is_mcpu_endpoint || sizeof(dma_addr_t) == 4 ||
+           dma_get_required_mask(&pdev->dev) <= 32) {
                 ioc->dma_mask = 32;
+               coherent_dma_mask = dma_mask = DMA_BIT_MASK(32);

That's not the upstream commit. The quoted upstream patch in the stable commit is:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e0e0747de0ea3dd87cdbb0393311e17471a9baf1

Which has the same diff as you quoted for the stable tree.

James

#5 lore raw
Quoted 31 lines · James Bottomley

On Mon, 2022-10-24 at 14:55 +0200, Juergen Gross wrote:
[...]

Quoted 21 lines

Please recheck the backported patch in 5.10.y. It is _wrong_. The backport
has:

drivers/scsi/mpt3sas/mpt3sas_base.c +1 -1
@@ -2993,7 +2993,7 @@ _base_config_dma_addressing(struct
 MPT3SAS_ADAPTER *ioc,
 struct pci_dev *pdev)
 
          if (ioc->is_mcpu_endpoint ||
              sizeof(dma_addr_t) == 4 || ioc->use_32bit_dma ||
-           dma_get_required_mask(&pdev->dev) <= 32)
+           dma_get_required_mask(&pdev->dev) <= DMA_BIT_MASK(32))
                  ioc->dma_mask = 32;
          /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
          else if (ioc->hba_mpi_version_belonged > MPI2_VERSION)

While the upstream patch has:

drivers/scsi/mpt3sas/mpt3sas_base.c

Quoted 4 lines
drivers/scsi/mpt3sas/mpt3sas_base.c +3 -0
+       if (ioc->is_mcpu_endpoint || sizeof(dma_addr_t) == 4 ||
+           dma_get_required_mask(&pdev->dev) <= 32) {
                  ioc->dma_mask = 32;
+               coherent_dma_mask = dma_mask = DMA_BIT_MASK(32);

That's not the upstream commit. The quoted upstream patch in the stable commit is:

Quoted 3 lines

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e0e0747de0ea3dd87cdbb0393311e17471a9baf1

Which has the same diff as you quoted for the stable tree.

Hmm, something is fishy here.

Looking at the source in current HEAD of Linus' tree shows me:

static int
_base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)

{
        struct sysinfo s;
        u64 coherent_dma_mask, dma_mask;

        if (ioc->is_mcpu_endpoint || sizeof(dma_addr_t) == 4 ||
            dma_get_required_mask(&pdev->dev) <= 32) {
                ioc->dma_mask = 32;
                coherent_dma_mask = dma_mask = DMA_BIT_MASK(32);
        /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */

This doesn't look like commit e0e0747de0ea3dd has been applied.

git log drivers/scsi/mpt3sas/mpt3sas_base.c doesn't show it either.

git name-rev e0e0747de0ea3dd tells me it should be in v6.0-rc7.

Juergen

#6 lore raw
In reply to Juergen Gross
Quoted 52 lines · Juergen Gross
Quoted 30 lines · Sreekanth Reddy
Quoted 7 lines · Bart Van Assche
Quoted 4 lines · Salvatore Bonaccorso

We got the following report in Debian after an update from 5.10.140 to the current 5.10.149. Full quoting below (from
https://bugs.debian.org/1022126). Does this ring some bell about known regressions?

Quoted 8 lines

Only three mpt3sas changes are new in v5.10.149 compared to v5.10.140:

$ git log --format=oneline v5.10.140..v5.10.149
2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()
e7fafef9830c4a01e60f76e3860a9bef0262378d scsi: mpt3sas: Force PCIe scatterlist allocations to be within same 4 GB region
ea10a652ad2ae2cf3eced6f632a5c98f26727057 scsi: mpt3sas: Fix use-after-free warning

Sreekanth and Suganath, can you help with bisecting this issue? For the full report, see also https://lore.kernel.org/linux-scsi/[email protected]/.

This issue is getting observed after having the below patch changes, 2b9aba0c5d58e141e32bb1bb4c7cd91d19f075b8 scsi: mpt3sas: Fix return value check of dma_get_required_mask()

What is happening is that on Xen hypervisor, this dma_get_required_mask() API always returns a 32 bit DMA mask. I.e. It says that the minimum DMA mask required to access the host memory is 32 bit and hence mpt3sas driver is setting the DMA mask to 32bit. So, on a 64 bit machine, if the driver set's the DMA mask to 32 bit then SWIOTLB's bounce buffer comes into picture during IOs. Since these bounce buffers are limited in size and hence we observe the IO hang if the large IOs are issued.

Quoted 4 lines

I am not sure whether this API's return value is correct or not in the Xen environment. If it is correct then I have to modify the driver to not use this API and directly set the DMA mask to 64 bit if the system is a 64bit machine.

Please recheck the backported patch in 5.10.y. It is _wrong_. The backport has:

drivers/scsi/mpt3sas/mpt3sas_base.c +1 -1
@@ -2993,7 +2993,7 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, 
 struct pci_dev *pdev)
 
         if (ioc->is_mcpu_endpoint ||
             sizeof(dma_addr_t) == 4 || ioc->use_32bit_dma ||
-           dma_get_required_mask(&pdev->dev) <= 32)
+           dma_get_required_mask(&pdev->dev) <= DMA_BIT_MASK(32))
                 ioc->dma_mask = 32;
         /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */

        else if (ioc->hba_mpi_version_belonged > MPI2_VERSION)

Quoted 6 lines

While the upstream patch has:

+       if (ioc->is_mcpu_endpoint || sizeof(dma_addr_t) == 4 || +           dma_get_required_mask(&pdev->dev) <= 32) {
                ioc->dma_mask = 32;
+               coherent_dma_mask = dma_mask = DMA_BIT_MASK(32);

Sorry for this mistake of mine, which seems to have been caused by a git inconsistency, as the upstream source is still showing the line

dma_get_required_mask(&pdev->dev) <= 32

I didn't double check which upstream patch was referenced by the backport patch, but looked at the output of "git blame" to look at the last patch older than the backport changing the line in question.

I didn't even think of the possibility that git could be wrong.

Juergen

#12 lore raw
In reply to Andrew Cooper
Quoted 10 lines · Andrew Cooper
Quoted 4 lines

I don't know exactly how this translates to Linux internals, but most devices are fine and it's routinely the mpt2/3sas drivers which encounter problems. It would be lovely if we could get to the bottom of this for once and for all.

So to summarize my two mails: I think te use of dma_get_required_mask in mpt3sas is wrong, and the dma_get_required_mask return value from xen-swiotlb is also wrong. Fixing either one should fix this problem, and I think we should fix both.

Hi all,

Is there anything that can be done to get this fixed in xen and the kernel somehow?
I keep having to manually patch the mpt3sas driver to prevent it from being able to switch to 32bit mode each time I upgrade the kernel.

Many thanks,

Joost

#13 lore raw
Quoted 11 lines · Joost Roeleveld
Quoted 10 lines · Andrew Cooper
Quoted 4 lines

I don't know exactly how this translates to Linux internals, but most devices are fine and it's routinely the mpt2/3sas drivers which encounter problems.  It would be lovely if we could get to the bottom of this for once and for all.

So to summarize my two mails: I think te use of dma_get_required_mask in mpt3sas is wrong, and the dma_get_required_mask return value from xen-swiotlb is also wrong. Fixing either one should fix this problem, and I think we should fix both.

Quoted 4 lines

Hi all,

Is there anything that can be done to get this fixed in xen and the kernel somehow?
I keep having to manually patch the mpt3sas driver to prevent it from being able to switch to 32bit mode each time I upgrade the kernel.

Just use 5.10.173.

Juergen

#14 lore raw
Quoted 13 lines · Juergen Gross
Quoted 12 lines · Joost Roeleveld
Quoted 11 lines · Andrew Cooper
Quoted 5 lines

I don't know exactly how this translates to Linux internals, but most devices are fine and it's routinely the mpt2/3sas drivers which encounter problems.  It would be lovely if we could get to the bottom of
this for once and for all.

So to summarize my two mails: I think te use of dma_get_required_mask in mpt3sas is wrong, and the dma_get_required_mask return value from xen-swiotlb is also wrong. Fixing either one should fix this problem, and I think we should fix both.

Quoted 8 lines
Quoted 6 lines

Hi all,

Is there anything that can be done to get this fixed in xen and the kernel somehow?
I keep having to manually patch the mpt3sas driver to prevent it from being able to switch to 32bit mode each time I upgrade the kernel.

Just use 5.10.173.

Its still broken even in 6.1

We've just stumbled over this yet again in XenServer.

~Andrew

#15 lore raw

Hi Andrew,

Quoted 21 lines · Andrew Cooper
Quoted 20 lines · Juergen Gross
Quoted 18 lines · Joost Roeleveld
Quoted 11 lines · Andrew Cooper
Quoted 5 lines

I don't know exactly how this translates to Linux internals, but most devices are fine and it's routinely the mpt2/3sas drivers which encounter problems.  It would be lovely if we could get to the bottom of
this for once and for all.

So to summarize my two mails: I think te use of dma_get_required_mask in mpt3sas is wrong, and the dma_get_required_mask return value from xen-swiotlb is also wrong. Fixing either one should fix this problem, and I think we should fix both.

Hi all,

Is there anything that can be done to get this fixed in xen and the kernel somehow?
I keep having to manually patch the mpt3sas driver to prevent it from being able to switch to 32bit mode each time I upgrade the kernel.

Quoted 5 lines
Quoted 1 line

Just use 5.10.173.

Its still broken even in 6.1

We've just stumbled over this yet again in XenServer.

The mentioned commit has been backported to 6.1.16, 5.15.100 and 5.10.173. Is the issue still reproducible with 6.1.16?

Regards,
Salvatore

#16 lore raw
In reply to Andrew Cooper
Quoted 24 lines

Hi Andrew,

Quoted 21 lines · Andrew Cooper
Quoted 20 lines · Juergen Gross
Quoted 18 lines · Joost Roeleveld
Quoted 11 lines · Andrew Cooper
Quoted 5 lines

I don't know exactly how this translates to Linux internals, but most devices are fine and it's routinely the mpt2/3sas drivers which encounter problems. It would be lovely if we could get to the bottom of
this for once and for all.

So to summarize my two mails: I think te use of dma_get_required_mask in mpt3sas is wrong, and the dma_get_required_mask return value from xen-swiotlb is also wrong. Fixing either one should fix this problem, and I think we should fix both.

Hi all,

Is there anything that can be done to get this fixed in xen and the kernel somehow?
I keep having to manually patch the mpt3sas driver to prevent it from being able to switch to 32bit mode each time I upgrade the kernel.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Just use 5.10.173.

Its still broken even in 6.1

We've just stumbled over this yet again in XenServer.

The mentioned commit has been backported to 6.1.16, 5.15.100 and 5.10.173. Is the issue still reproducible with 6.1.16?

I ran into this on 6.1.12.

Not in a position to test with 6.1.16+ yet as the only system using this driver is production.

Kind regards,

Joost