SWIOTLB on 32-bit PAE
Hi,
I discovered that my 32-bit PAE 4.2.0 kernel (no IOMMU code) would hang when writing to my USB disk. The kernel spews million(-ish messages per sec) to syslog, effectively "hanging" userspace with my kernel.
Oct 2 14:33:06 voodoochild kernel: [ 223.287447] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff Oct 2 14:33:06 voodoochild kernel: [ 223.287448] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff Oct 2 14:33:06 voodoochild kernel: [ 223.287449] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff
... etc ...
In my kernel config I noticed that SWIOTLB was not on. It seems SWIOTLB is provided for 64-bit and 32-bit with IOMMU/AGPGART code. But if I compiled the kernel with PAE and no IOMMU and no other GART code, I would not get SWIOTLB. I'd like to think that SWIOTLB should be selected for 32-bit PAE as default.
I have attached a oneliner patch which does that.
The patch works for me. The issue where the kernel more or less runs endless bashing of (nommu_?)map_sg when failing is another problem I guess. I expected the kernel/drivers to have some more graceful handling, but I know to little about this area to have a proper opinion.
Regards,
Christian
arch/x86/Kconfig +1 -0
@@ -1282,6 +1282,7 @@ config X86_PAE bool "PAE (Physical Address Extension) Support" depends on X86_32 && !HIGHMEM4G + select SWIOTLB ---help--- PAE is required for NX support, and furthermore enables larger swapspace support for non-overcommit purposes. It
Quoted 10 lines · Christian Melki
Joerg,
I already sent the patch for
Kconfigchange to the list.
But I discovered something even more disturbing which I mailed Konrad about. With the change my machine does not hang any more, but instead corrupts data which is written. The data however is readable on the machine which corrupted it, unreadable on the other machine without the patch. I don't understand what is going on. I have never encountered "selective" file corruption of fs on different machines. The fs itself checks out cleanly with badblock control (ext4). I fail to understand what is happening.
And I believe this ended up being an operator error?
With the patch you sent, you didn't include your SoB (Signed-off-By). I would like to include your patch but I cannot with your SoB. It is matter of reading to Documentation/SubmittingPatches and just consenting to the Certificate in there.
If you are uncomfortable with that - that is OK, I can make this patch myself (a one liner :-)) but I figured it owuld be nice for you to have all the credit.
Quoted 55 lines
Regards,
Christian-------- Forwarded Message --------
Subject: SWIOTLB on 32-bit PAE
Date: Mon, 5 Oct 2015 10:31:39 +0200
From: Christian Melki <[email protected]>
To: [email protected]
CC: [email protected]Hi,
I discovered that my 32-bit PAE 4.2.0 kernel (no IOMMU code) would hang when writing to my USB disk. The kernel spews million(-ish messages per sec) to syslog, effectively "hanging" userspace with my kernel.
Oct 2 14:33:06 voodoochild kernel: [ 223.287447] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff Oct 2 14:33:06 voodoochild kernel: [ 223.287448] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff Oct 2 14:33:06 voodoochild kernel: [ 223.287449] nommu_map_sg:overflow 25dcac000+1024 of device mask ffffffff
... etc ...In my kernel config I noticed that SWIOTLB was not on. It seems SWIOTLB is provided for 64-bit and 32-bit with IOMMU/AGPGART code. But if I compiled the kernel with PAE and no IOMMU and no other GART code, I would not get SWIOTLB. I'd like to think that SWIOTLB should be selected for 32-bit PAE as default.
I have attached a oneliner patch which does that.
The patch works for me. The issue where the kernel more or less runs endless bashing of (nommu_?)map_sg when failing is another problem I guess. I expected the kernel/drivers to have some more graceful handling, but I know to little about this area to have a proper opinion.Regards,
Christiandiff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
arch/x86/Kconfig +1
-0@@ -1282,6 +1282,7 @@ config X86_PAE bool "PAE (Physical Address Extension) Support" depends on X86_32 && !HIGHMEM4G + select SWIOTLB ---help--- PAE is required for NX support, and furthermore enables larger swapspace support for non-overcommit purposes. It
Yes. Operator error indeed.
Thanks for asking. Included is the same patch with the SoB.
Quoted 65 lines · Konrad Rzeszutek Wilk
Quoted 10 lines · Christian Melki
Joerg,
I already sent the patch for
Kconfigchange to the list.
But I discovered something even more disturbing which I mailed Konrad about. With the change my machine does not hang any more, but instead corrupts data which is written. The data however is readable on the machine which corrupted it, unreadable on the other machine without the patch. I don't understand what is going on. I have never encountered "selective" file corruption of fs on different machines. The fs itself checks out cleanly with badblock control (ext4). I fail to understand what is happening.And I believe this ended up being an operator error?
With the patch you sent, you didn't include your SoB (Signed-off-By). I would like to include your patch but I cannot with your SoB. It is matter of reading to
Documentation/SubmittingPatchesand just consenting to the Certificate in there.If you are uncomfortable with that - that is OK, I can make this patch myself (a one liner :-)) but I figured it owuld be nice for you to have all the credit.
Quoted 42 lines
Regards,
Christian-------- Forwarded Message --------
Subject: SWIOTLB on 32-bit PAE
Date: Mon, 5 Oct 2015 10:31:39 +0200
From: Christian Melki <[email protected]>
To: [email protected]
CC: [email protected]Hi,
I discovered that my 32-bit PAE 4.2.0 kernel (no IOMMU code) would hang when writing to my USB disk. The kernel spews million(-ish messages per sec) to syslog, effectively "hanging" userspace with my kernel.
Oct 2 14:33:06 voodoochild kernel: [ 223.287447] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff Oct 2 14:33:06 voodoochild kernel: [ 223.287448] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff Oct 2 14:33:06 voodoochild kernel: [ 223.287449] nommu_map_sg:overflow 25dcac000+1024 of device mask ffffffff
... etc ...In my kernel config I noticed that SWIOTLB was not on. It seems SWIOTLB is provided for 64-bit and 32-bit with IOMMU/AGPGART code. But if I compiled the kernel with PAE and no IOMMU and no other GART code, I would not get SWIOTLB. I'd like to think that SWIOTLB should be selected for 32-bit PAE as default.
I have attached a oneliner patch which does that.
The patch works for me. The issue where the kernel more or less runs endless bashing of (nommu_?)map_sg when failing is another problem I guess. I expected the kernel/drivers to have some more graceful handling, but I know to little about this area to have a proper opinion.Regards,
Christian
Quoted 13 lines
arch/x86/Kconfig line 1288
Quoted 12 lines
diff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
arch/x86/Kconfig +1
-0@@ -1282,6 +1282,7 @@ config X86_PAE bool "PAE (Physical Address Extension) Support" depends on X86_32 && !HIGHMEM4G + select SWIOTLB ---help--- PAE is required for NX support, and furthermore enables larger swapspace support for non-overcommit purposes. It
arch/x86/Kconfig +1 -0
@@ -1282,6 +1282,7 @@ config X86_PAE bool "PAE (Physical Address Extension) Support" depends on X86_32 && !HIGHMEM4G + select SWIOTLB ---help--- PAE is required for NX support, and furthermore enables larger swapspace support for non-overcommit purposes. It
Quoted 3 lines · Christian Melki
Yes. Operator error indeed.
Thanks for asking. Included is the same patch with the SoB.
I've stuck it at
https://git.kernel.org/cgit/linux/kernel/git/konrad/swiotlb.git/ #stable/for-linus-4.3
Will send it out tomorrow if there are no #linux-next issues.
Quoted 95 lines · Konrad Rzeszutek Wilk
Quoted 65 lines · Christian Melki
Quoted 10 lines
Joerg,
I already sent the patch for
Kconfigchange to the list.
But I discovered something even more disturbing which I mailed Konrad about. With the change my machine does not hang any more, but instead corrupts data which is written. The data however is readable on the machine which corrupted it, unreadable on the other machine without the patch. I don't understand what is going on. I have never encountered "selective" file corruption of fs on different machines. The fs itself checks out cleanly with badblock control (ext4). I fail to understand what is happening.And I believe this ended up being an operator error?
With the patch you sent, you didn't include your SoB (Signed-off-By). I would like to include your patch but I cannot with your SoB. It is matter of reading to
Documentation/SubmittingPatchesand just consenting to the Certificate in there.If you are uncomfortable with that - that is OK, I can make this patch myself (a one liner :-)) but I figured it owuld be nice for you to have all the credit.
Quoted 42 lines
Regards,
Christian-------- Forwarded Message --------
Subject: SWIOTLB on 32-bit PAE
Date: Mon, 5 Oct 2015 10:31:39 +0200
From: Christian Melki <[email protected]>
To: [email protected]
CC: [email protected]Hi,
I discovered that my 32-bit PAE 4.2.0 kernel (no IOMMU code) would hang when writing to my USB disk. The kernel spews million(-ish messages per sec) to syslog, effectively "hanging" userspace with my kernel.
Oct 2 14:33:06 voodoochild kernel: [ 223.287447] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff Oct 2 14:33:06 voodoochild kernel: [ 223.287448] nommu_map_sg: overflow 25dcac000+1024 of device mask ffffffff Oct 2 14:33:06 voodoochild kernel: [ 223.287449] nommu_map_sg:overflow 25dcac000+1024 of device mask ffffffff
... etc ...In my kernel config I noticed that SWIOTLB was not on. It seems SWIOTLB is provided for 64-bit and 32-bit with IOMMU/AGPGART code. But if I compiled the kernel with PAE and no IOMMU and no other GART code, I would not get SWIOTLB. I'd like to think that SWIOTLB should be selected for 32-bit PAE as default.
I have attached a oneliner patch which does that.
The patch works for me. The issue where the kernel more or less runs endless bashing of (nommu_?)map_sg when failing is another problem I guess. I expected the kernel/drivers to have some more graceful handling, but I know to little about this area to have a proper opinion.Regards,
ChristianQuoted 13 lines
arch/x86/Kconfig line 1288
Quoted 12 lines
diff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
arch/x86/Kconfig +1
-0@@ -1282,6 +1282,7 @@ config X86_PAE bool "PAE (Physical Address Extension) Support" depends on X86_32 && !HIGHMEM4G + select SWIOTLB ---help--- PAE is required for NX support, and furthermore enables larger swapspace support for non-overcommit purposes. Itdiff -urN linux-4.2.orig/arch/x86/Kconfig linux-4.2/arch/x86/Kconfig
arch/x86/Kconfig +1
-0@@ -1282,6 +1282,7 @@ config X86_PAE bool "PAE (Physical Address Extension) Support" depends on X86_32 && !HIGHMEM4G + select SWIOTLB ---help--- PAE is required for NX support, and furthermore enables larger swapspace support for non-overcommit purposes. ItSigned-off-by Christian Melki <[email protected]>