riscv: enable THP_SWAP for RV64
PATCH v2 1/1 riscv: enable THP_SWAP for RV64
I have a Sipeed Lichee RV dock board which only has 512MB DDR, so memory optimizations such as swap on zram are helpful. As is seen in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after swapped out"), THP_SWAP can improve the swap throughput significantly.
Enable THP_SWAP for RV64, testing the micro-benchmark which is introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") shows below numbers on the Lichee RV dock board:
thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests) thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
Improved by 382%!
---
Since v1:
- collect reviewed-by tag
- make ARCH_WANTS_THP_SWAP rely on HAVE_ARCH_TRANSPARENT_HUGEPAGE instead
arch/riscv/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/riscv/Kconfig +1 -0
@@ -45,6 +45,7 @@ config RISCV select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE if 64BIT + select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLONE_BACKWARDS
Signature · 1 line
2.34.1
Hey Jisheng,
Quoted 6 lines · Jisheng Zhang
I have a Sipeed Lichee RV dock board which only has 512MB DDR, so memory optimizations such as swap on zram are helpful. As is seen in commit
d0637c505f8a("arm64: enableTHP_SWAPfor arm64") and commitbd4c82c22c367e("mm, THP, swap: delay splitting THP after swapped out"),THP_SWAPcan improve the swap throughput significantly.
Quoted 6 lines
Enable
THP_SWAPfor RV64, testing the micro-benchmark which is introduced by commitd0637c505f8a("arm64: enableTHP_SWAPfor arm64") shows below numbers on the Lichee RV dock board:thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests) thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
I know the original commit message contains this, but it's a little odd. If the patch /enables/ THP then how would there be THP swap prior to the patch?
Quoted 2 lines
Improved by 382%!
I could not replicate the after numbers on my nezha, so I suspect I am missing something in my config/setup. zswap is enabled and is working, TRANSPARENT_HUGEPAGE is enabled etc. Not that it matters for acceptance of the patch though.
I gave it a try and nothing went up in flames while using zswap so:
Quoted 24 lines
Signed-off-by Jisheng Zhang <[email protected]>Reviewed-by Andrew Jones <[email protected]>---
Since v1:
- collect reviewed-by tag
- makeARCH_WANTS_THP_SWAPrely onHAVE_ARCH_TRANSPARENT_HUGEPAGEinsteadarch/riscv/Kconfig | 1 + 1 file changed, 1 insertion(+)arch/riscv/Kconfig +1
-0@@ -45,6 +45,7 @@ config RISCV select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE if 64BIT + select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLONE_BACKWARDS
Quoted 1 line · On Sat, Aug 27, 2022 at 09:13:03PM +0000, [email protected] wrote:
Hey Jisheng,
Hi Conor,
Quoted 11 lines · Jisheng Zhang
Quoted 10 lines
I have a Sipeed Lichee RV dock board which only has 512MB DDR, so memory optimizations such as swap on zram are helpful. As is seen in commit
d0637c505f8a("arm64: enableTHP_SWAPfor arm64") and commitbd4c82c22c367e("mm, THP, swap: delay splitting THP after swapped out"),THP_SWAPcan improve the swap throughput significantly.Enable
THP_SWAPfor RV64, testing the micro-benchmark which is introduced by commitd0637c505f8a("arm64: enableTHP_SWAPfor arm64") shows below numbers on the Lichee RV dock board:
Quoted 6 lines
Quoted 2 lines
thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests) thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
I know the original commit message contains this, but it's a little odd. If the patch /enables/ THP then how would there be THP swap prior to the patch?
hmm, it's swap I'll send a v3 to correct the description.
Quoted 6 lines
Quoted 3 lines
Improved by 382%!
I could not replicate the after numbers on my nezha, so I suspect I am missing something in my config/setup. zswap is enabled and is
swap on zram rather than zswap ;)
Quoted 30 lines
working,
TRANSPARENT_HUGEPAGEis enabled etc. Not that it matters for acceptance of the patch though.I gave it a try and nothing went up in flames while using zswap so:
Reviewed-by Conor Dooley <[email protected]>Quoted 24 lines
Signed-off-by Jisheng Zhang <[email protected]>Reviewed-by Andrew Jones <[email protected]>---
Since v1:
- collect reviewed-by tag
- makeARCH_WANTS_THP_SWAPrely onHAVE_ARCH_TRANSPARENT_HUGEPAGEinsteadarch/riscv/Kconfig | 1 + 1 file changed, 1 insertion(+)arch/riscv/Kconfig +1
-0@@ -45,6 +45,7 @@ config RISCV select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE if 64BIT + select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLONE_BACKWARDS
Quoted 29 lines · Jisheng Zhang
EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
Quoted 1 line · On Sat, Aug 27, 2022 at 09:13:03PM +0000, [email protected] wrote:
Hey Jisheng,
Hi Conor,
Quoted 11 lines · Jisheng Zhang
Quoted 10 lines
I have a Sipeed Lichee RV dock board which only has 512MB DDR, so memory optimizations such as swap on zram are helpful. As is seen in commit
d0637c505f8a("arm64: enableTHP_SWAPfor arm64") and commitbd4c82c22c367e("mm, THP, swap: delay splitting THP after swapped out"),THP_SWAPcan improve the swap throughput significantly.Enable
THP_SWAPfor RV64, testing the micro-benchmark which is introduced by commitd0637c505f8a("arm64: enableTHP_SWAPfor arm64") shows below numbers on the Lichee RV dock board:Quoted 6 lines
Quoted 2 lines
thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests) thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
I know the original commit message contains this, but it's a little odd. If the patch /enables/ THP then how would there be THP swap prior to the patch?
hmm, it's swap I'll send a v3 to correct the description.
Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Improved by 382%!
I could not replicate the after numbers on my nezha, so I suspect I am missing something in my config/setup. zswap is enabled and is
swap on zram rather than zswap ;)
I think I tried about 30 different config variations, initially not using zswap and later using it.
My zramctl looks like so (although I did try zstd too) after running the demo application from that commit:
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT/dev/zram0 lzo-rle 241M 22M 8.4M 9.1M 1 [SWAP]
I am using the default riscv defconfig + the following:CONFIG_ZRAM=yCONFIG_CRYPTO_DEFLATE=yCONFIG_CRYPTO_LZO=yCONFIG_CRYPTO_ZSTD=y CONFIG_ZRAM_MEMORY_TRACKING=y CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y CONFIG_THP_SWAP=y
Am I just missing something obvious here?
Sorry,
Conor.
Quoted 55 lines · On Mon, Aug 29, 2022 at 05:27:48PM +0000, [email protected] wrote:
Quoted 29 lines · Jisheng Zhang
EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
Quoted 1 line · On Sat, Aug 27, 2022 at 09:13:03PM +0000, [email protected] wrote:
Hey Jisheng,
Hi Conor,
Quoted 11 lines · Jisheng Zhang
Quoted 10 lines
I have a Sipeed Lichee RV dock board which only has 512MB DDR, so memory optimizations such as swap on zram are helpful. As is seen in commit
d0637c505f8a("arm64: enableTHP_SWAPfor arm64") and commitbd4c82c22c367e("mm, THP, swap: delay splitting THP after swapped out"),THP_SWAPcan improve the swap throughput significantly.Enable
THP_SWAPfor RV64, testing the micro-benchmark which is introduced by commitd0637c505f8a("arm64: enableTHP_SWAPfor arm64") shows below numbers on the Lichee RV dock board:Quoted 6 lines
Quoted 2 lines
thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests) thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
I know the original commit message contains this, but it's a little odd. If the patch /enables/ THP then how would there be THP swap prior to the patch?
hmm, it's swap I'll send a v3 to correct the description.
Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Improved by 382%!
I could not replicate the after numbers on my nezha, so I suspect I am missing something in my config/setup. zswap is enabled and is
swap on zram rather than zswap ;)
I think I tried about 30 different config variations, initially not using zswap and later using it.
My zramctl looks like so (although I did try zstd too) after running the demo application from that commit:NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0lzo-rle 241M 22M 8.4M 9.1M 1 [SWAP]I am using the default riscv defconfig + the following:
CONFIG_ZRAM=yCONFIG_CRYPTO_DEFLATE=yCONFIG_CRYPTO_LZO=yCONFIG_CRYPTO_ZSTD=yCONFIG_ZRAM_MEMORY_TRACKING=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_MADVISE=yCONFIG_THP_SWAP=y
Quoted 1 line
Am I just missing something obvious here?
similar config options here. what's your rootfs? Is your board busy with something? I used a minimal rootfs built from buildroot. can you plz show your numbers w/ and w/o the patch?
I also tried the simple benchmark on qemu(just for reference, since I have no other riscv boards except the lichee RV dock board):
swp out w/o patch: 30066 bytes/ms (mean of 10 tests)
swp out w/ patch: 130055 bytes/ms (mean of 10 tests)
so improved by 332.7%
Quoted 56 lines · Jisheng Zhang
Quoted 55 lines · On Mon, Aug 29, 2022 at 05:27:48PM +0000, [email protected] wrote:
Quoted 29 lines · Jisheng Zhang
EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
Quoted 1 line · On Sat, Aug 27, 2022 at 09:13:03PM +0000, [email protected] wrote:
Hey Jisheng,
Hi Conor,
Quoted 11 lines · Jisheng Zhang
Quoted 10 lines
I have a Sipeed Lichee RV dock board which only has 512MB DDR, so memory optimizations such as swap on zram are helpful. As is seen in commit
d0637c505f8a("arm64: enableTHP_SWAPfor arm64") and commitbd4c82c22c367e("mm, THP, swap: delay splitting THP after swapped out"),THP_SWAPcan improve the swap throughput significantly.Enable
THP_SWAPfor RV64, testing the micro-benchmark which is introduced by commitd0637c505f8a("arm64: enableTHP_SWAPfor arm64") shows below numbers on the Lichee RV dock board:Quoted 6 lines
Quoted 2 lines
thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests) thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
I know the original commit message contains this, but it's a little odd. If the patch /enables/ THP then how would there be THP swap prior to the patch?
hmm, it's swap I'll send a v3 to correct the description.
Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Improved by 382%!
I could not replicate the after numbers on my nezha, so I suspect I am missing something in my config/setup. zswap is enabled and is
swap on zram rather than zswap ;)
I think I tried about 30 different config variations, initially not using zswap and later using it.
My zramctl looks like so (although I did try zstd too) after running the demo application from that commit:NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0lzo-rle 241M 22M 8.4M 9.1M 1 [SWAP]I am using the default riscv defconfig + the following:
CONFIG_ZRAM=yCONFIG_CRYPTO_DEFLATE=yCONFIG_CRYPTO_LZO=yCONFIG_CRYPTO_ZSTD=yCONFIG_ZRAM_MEMORY_TRACKING=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_MADVISE=yCONFIG_THP_SWAP=y
Quoted 5 lines
Quoted 1 line
Am I just missing something obvious here?
similar config options here. what's your rootfs? Is your board busy with something? I used a minimal rootfs built from buildroot. can you plz show your numbers w/ and w/o the patch?
I was using fedora for the testing, downloaded directly from koji. My before/after numbers varied, but were around 80,000 bytes/ms most of the time.
If I increased the size to 500 * 1024 * 1024 I got around 130k.
Before/after the patch, the numbers did not really change, but things did fluctuate quite wildly - from about 50k to 90k using the 400 size.
Quoted 6 lines
I also tried the simple benchmark on
qemu(just for reference, since I have no other riscv boards except the lichee RV dock board):
swp out w/o patch: 30066 bytes/ms (mean of 10 tests)
swp out w/ patch: 130055 bytes/ms (mean of 10 tests)
so improved by 332.7%
I'll give QEMU a go so :)
Quoted 63 lines · On 30/08/2022 15:15, Conor Dooley - M52691 wrote:
Quoted 56 lines · Jisheng Zhang
Quoted 55 lines · On Mon, Aug 29, 2022 at 05:27:48PM +0000, [email protected] wrote:
Quoted 29 lines · Jisheng Zhang
EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
Quoted 1 line · On Sat, Aug 27, 2022 at 09:13:03PM +0000, [email protected] wrote:
Hey Jisheng,
Hi Conor,
Quoted 11 lines · Jisheng Zhang
Quoted 10 lines
I have a Sipeed Lichee RV dock board which only has 512MB DDR, so memory optimizations such as swap on zram are helpful. As is seen in commit
d0637c505f8a("arm64: enableTHP_SWAPfor arm64") and commitbd4c82c22c367e("mm, THP, swap: delay splitting THP after swapped out"),THP_SWAPcan improve the swap throughput significantly.Enable
THP_SWAPfor RV64, testing the micro-benchmark which is introduced by commitd0637c505f8a("arm64: enableTHP_SWAPfor arm64") shows below numbers on the Lichee RV dock board:Quoted 6 lines
Quoted 2 lines
thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests) thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
I know the original commit message contains this, but it's a little odd. If the patch /enables/ THP then how would there be THP swap prior to the patch?
hmm, it's swap I'll send a v3 to correct the description.
Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Improved by 382%!
I could not replicate the after numbers on my nezha, so I suspect I am missing something in my config/setup. zswap is enabled and is
swap on zram rather than zswap ;)
I think I tried about 30 different config variations, initially not using zswap and later using it.
My zramctl looks like so (although I did try zstd too) after running the demo application from that commit:NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0lzo-rle 241M 22M 8.4M 9.1M 1 [SWAP]I am using the default riscv defconfig + the following:
CONFIG_ZRAM=yCONFIG_CRYPTO_DEFLATE=yCONFIG_CRYPTO_LZO=yCONFIG_CRYPTO_ZSTD=yCONFIG_ZRAM_MEMORY_TRACKING=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_MADVISE=yCONFIG_THP_SWAP=yQuoted 5 lines
Quoted 1 line
Am I just missing something obvious here?
similar config options here. what's your rootfs? Is your board busy with something? I used a minimal rootfs built from buildroot. can you plz show your numbers w/ and w/o the patch?
Quoted 5 lines
I was using fedora for the testing, downloaded directly from koji. My before/after numbers varied, but were around 80,000 bytes/ms most of the time.
If I increased the size to 500 * 1024 * 1024 I got around 130k.
130k before & after.**
Quoted 4 lines
Before/after the patch, the numbers did not really change, but things did fluctuate quite wildly - from about 50k to 90k using the 400 size.
What I mean is: before/after the patch had visible performance difference because it was always flucuating in the same range.
Quoted 9 lines
Quoted 7 lines
I also tried the simple benchmark on
qemu(just for reference, since I have no other riscv boards except the lichee RV dock board):
swp out w/o patch: 30066 bytes/ms (mean of 10 tests)
swp out w/ patch: 130055 bytes/ms (mean of 10 tests)
so improved by 332.7%I'll give QEMU a go so :)
Quoted 73 lines · On Tue, Aug 30, 2022 at 02:26:38PM +0000, [email protected] wrote:
Quoted 63 lines · On 30/08/2022 15:15, Conor Dooley - M52691 wrote:
Quoted 56 lines · Jisheng Zhang
Quoted 55 lines · On Mon, Aug 29, 2022 at 05:27:48PM +0000, [email protected] wrote:
Quoted 29 lines · Jisheng Zhang
EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
Quoted 1 line · On Sat, Aug 27, 2022 at 09:13:03PM +0000, [email protected] wrote:
Hey Jisheng,
Hi Conor,
Quoted 11 lines · Jisheng Zhang
Quoted 10 lines
I have a Sipeed Lichee RV dock board which only has 512MB DDR, so memory optimizations such as swap on zram are helpful. As is seen in commit
d0637c505f8a("arm64: enableTHP_SWAPfor arm64") and commitbd4c82c22c367e("mm, THP, swap: delay splitting THP after swapped out"),THP_SWAPcan improve the swap throughput significantly.Enable
THP_SWAPfor RV64, testing the micro-benchmark which is introduced by commitd0637c505f8a("arm64: enableTHP_SWAPfor arm64") shows below numbers on the Lichee RV dock board:Quoted 6 lines
Quoted 2 lines
thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests) thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
I know the original commit message contains this, but it's a little odd. If the patch /enables/ THP then how would there be THP swap prior to the patch?
hmm, it's swap I'll send a v3 to correct the description.
Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Improved by 382%!
I could not replicate the after numbers on my nezha, so I suspect I am missing something in my config/setup. zswap is enabled and is
swap on zram rather than zswap ;)
I think I tried about 30 different config variations, initially not using zswap and later using it.
My zramctl looks like so (although I did try zstd too) after running the demo application from that commit:NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0lzo-rle 241M 22M 8.4M 9.1M 1 [SWAP]I am using the default riscv defconfig + the following:
CONFIG_ZRAM=yCONFIG_CRYPTO_DEFLATE=yCONFIG_CRYPTO_LZO=yCONFIG_CRYPTO_ZSTD=yCONFIG_ZRAM_MEMORY_TRACKING=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_MADVISE=yCONFIG_THP_SWAP=yQuoted 5 lines
Quoted 1 line
Am I just missing something obvious here?
similar config options here. what's your rootfs? Is your board busy with something? I used a minimal rootfs built from buildroot. can you plz show your numbers w/ and w/o the patch?
Quoted 5 lines
I was using fedora for the testing, downloaded directly from koji. My before/after numbers varied, but were around 80,000 bytes/ms most of the time.
If I increased the size to 500 * 1024 * 1024 I got around 130k.
130k before & after.**
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Before/after the patch, the numbers did not really change, but things did fluctuate quite wildly - from about 50k to 90k using the 400 size.
What I mean is: before/after the patch had visible performance difference because it was always flucuating in the same range.
I see the difference -- w/ minial buildroot rootfs, the numbers isn't kept the same but the difference is trivial, I even got two or three the same numbers during 10 times of testing. But your numbers were always flucuating, so I guess your system maybe busy with with something in a shore period, I.E the os env is full of noise.
I guess you may get similar improvement percentage when trying buildroot
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
I also tried the simple benchmark on
qemu(just for reference, since I have no other riscv boards except the lichee RV dock board):
swp out w/o patch: 30066 bytes/ms (mean of 10 tests)
swp out w/ patch: 130055 bytes/ms (mean of 10 tests)
so improved by 332.7%I'll give QEMU a go so :)