Skip to contentSkip to conversation

*Enable build of full Xen for RISC-V

PATCH Superseded Oleksii Kurochko 4 participants 37 messages May 29, 2024 – Jun 18, 2024 RISC-V

v12 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v12 cover *Enable build of full Xen for RISC-V

This patch series performs all of the additions necessary to drop the build overrides for RISCV and enable the full Xen build. Except in cases where compatibile implementations already exist (e.g. atomic.h and bitops.h), the newly added definitions are simple.

The patch series is based on the following patch series:
- [PATCH v2 for-4.19 00/13] xen/bitops: Untangle ffs()/fls() infrastructure [1]
- [PATCH] move __read_mostly to xen/cache.h [2]

Right now, the patch series doesn't have a direct dependency on [2] and it provides __read_mostly in the patch:
[PATCH v3 26/34] xen/riscv: add definition of __read_mostly
However, it will be dropped as soon as [2] is merged or at least when the final version of the patch [2] is provided.
As an option, it can be considered to merge arch-specific patch and then just rebase [2] and drop arch-specific changes.

The link to the branch with rebased patches on top of [1] could be found here:
https://gitlab.com/xen-project/people/olkur/xen/-/commits/riscv-full-xen-build-v12

[1] https://lore.kernel.org/xen-devel/[email protected]/
[2] https://lore.kernel.org/xen-devel/[email protected]/

***
I think that I need to write a separate email requesting approval for this patch series to be included in Xen 4.19. Most of the patches are RISC-V specific, so there is a low risk of breaking anything else.
There is only one patch that touches common or arch-specific files, and it doesn't introduce any functional changes.
Since I can't approve it myself, I am asking for someone else to do so.
***

---
Changes in V12:
- Rebase the patch series on top of [1] mentioned above.
- Update the cover letter message.
- "[PATCH v11 3/9] xen/bitops: implement fls{l}() in common logic" was droped as it is a part of patch series [1] mentioned above.
- Other changes are specific to specific patches. Please look at changes for specific patch.
---
Changes in V11:
- Patch was merged to staging:
- [PATCH v10 05/14] xen/riscv: introduce cmpxchg.h
- [PATCH v10 06/14] xen/riscv: introduce atomic.h
- [PATCH v10 07/14] xen/riscv: introduce monitor.h
- [PATCH v10 09/14] xen/riscv: add required things to current.h
- [PATCH v10 11/14] xen/riscv: introduce vm_event_*() functions
- Other changes are specific to specific patches. Please look at changes for specific patch.
---
Changes in V10:
- Patch was merged to staging:
- [PATCH v9 04/15] xen/bitops: put __ffs() into linux compatible header
- Other changes are specific to specific patches. Please look at changes for specific patch.
---
Changes in V9:
- Patch was merged to staging:
- [PATCH v8 07/17] xen/riscv: introduce io.h
- [PATCH v7 14/19] xen/riscv: add minimal stuff to page.h to build full Xen
- Other changes are specific to specific patches. Please look at changes for specific patch.
---
Changes in V8:
- Patch was merged to staging:
- [PATCH v7 01/19] automation: introduce fixed randconfig for RISC-V
- [PATCH v7 03/19] xen/riscv: introduce extenstion support check by compiler
- Other changes are specific to specific patches. Please look at changes for specific patch.
- Update the commit message:
- drop the dependency from STATIC_ASSERT_UNREACHABLE() implementation.
- Add suggestion to merge arch-specific changes related to __read_mostly.
---
Changes in V7:
- Patch was merged to staging:
[PATCH v6 15/20] xen/riscv: add minimal stuff to processor.h to build full Xen.
- Other changes are specific to specific patches. Please look at changes for specific patch.
---
Changes in V6:
- Update the cover letter message: drop already merged dependecies and add
a new one.
- Patches were merged to staging:
- [PATCH v5 02/23] xen/riscv: use some asm-generic headers ( even v4 was merged to staging branch, I just wasn't apply changes on top of the latest staging branch )
- [PATCH v5 03/23] xen/riscv: introduce nospec.h
- [PATCH v5 10/23] xen/riscv: introduces acrquire, release and full barriers
- Introduce new patches:
- xen/riscv: introduce extenstion support check by compiler
- xen/bitops: put __ffs() and ffz() into linux compatible header
- xen/bitops: implement fls{l}() in common logic
- The following patches were dropped:
- drop some patches related to bitops operations as they were introduced in another

     patch series [...]

- introduce new version for generic __ffs(), ffz() and fls{l}().
- Merge patch from patch series "[PATCH v9 0/7] Introduce generic headers" to this patch series as only one patch left in the generic headers patch series and it is more about RISC-V.
- Other changes are specific to specific patches. please look at specific patch.
---
Changes in V5:
- Update the cover letter as one of the dependencies were merged to staging.
- Was introduced asm-generic for atomic ops and separate patches for asm-generic bit ops
- Moved fence.h to separate patch to deal with some patches dependecies on fence.h
- Patches were dropped as they were merged to staging:
* [PATCH v4 03/30] xen: add support in public/hvm/save.h for PPC and RISC-V
* [PATCH v4 04/30] xen/riscv: introduce cpufeature.h
* [PATCH v4 05/30] xen/riscv: introduce guest_atomics.h
* [PATCH v4 06/30] xen: avoid generation of empty asm/iommu.h
* [PATCH v4 08/30] xen/riscv: introduce setup.h
* [PATCH v4 10/30] xen/riscv: introduce flushtlb.h
* [PATCH v4 11/30] xen/riscv: introduce smp.h
* [PATCH v4 15/30] xen/riscv: introduce irq.h
* [PATCH v4 16/30] xen/riscv: introduce p2m.h
* [PATCH v4 17/30] xen/riscv: introduce regs.h
* [PATCH v4 18/30] xen/riscv: introduce time.h
* [PATCH v4 19/30] xen/riscv: introduce event.h
* [PATCH v4 22/30] xen/riscv: define an address of frame table
- Other changes are specific to specific patches. please look at specific patch
---
Changes in V4:
- Update the cover letter message: new patch series dependencies.
- Some patches were merged to staging, so they were dropped in this patch series:
[PATCH v3 09/34] xen/riscv: introduce system.h
[PATCH v3 18/34] xen/riscv: introduce domain.h
[PATCH v3 19/34] xen/riscv: introduce guest_access.h
- Was sent out of this patch series:
[PATCH v3 16/34] xen/lib: introduce generic find next bit operations
- [PATCH v3 17/34] xen/riscv: add compilation of generic find-next-bit.c was droped as CONFIG_GENERIC_FIND_NEXT_BIT was dropped.
- All other changes are specific to a specific patch.
---
Changes in V3:
- Update the cover letter message
- The following patches were dropped as they were merged to staging:
[PATCH v2 03/39] xen/riscv:introduce asm/byteorder.h
[PATCH v2 04/39] xen/riscv: add public arch-riscv.h
[PATCH v2 05/39] xen/riscv: introduce spinlock.h
[PATCH v2 20/39] xen/riscv: define bug frame tables in xen.lds.S
[PATCH v2 34/39] xen: add RISCV support for pmu.h
[PATCH v2 35/39] xen: add necessary headers to common to build full Xen for RISC-V
- Instead of the following patches were introduced new:
[PATCH v2 10/39] xen/riscv: introduce asm/iommu.h
[PATCH v2 11/39] xen/riscv: introduce asm/nospec.h
- remove "asm/" for commit messages which start with "xen/riscv:"
- code style updates.
- add emulation of {cmp}xchg_* for 1 and 2 bytes types.
- code style fixes.
- add SPDX and footer for the newly added headers.
- introduce generic find-next-bit.c.
- some other mionor changes. ( details please find in a patch )
---
Changes in V2:
- Drop the following patches as they are the part of [2]:
[PATCH v1 06/57] xen/riscv: introduce paging.h
[PATCH v1 08/57] xen/riscv: introduce asm/device.h
[PATCH v1 10/57] xen/riscv: introduce asm/grant_table.h
[PATCH v1 12/57] xen/riscv: introduce asm/hypercall.h
[PATCH v1 13/57] xen/riscv: introduce asm/iocap.h
[PATCH v1 15/57] xen/riscv: introduce asm/mem_access.h
[PATCH v1 18/57] xen/riscv: introduce asm/random.h
[PATCH v1 21/57] xen/riscv: introduce asm/xenoprof.h
[PATCH v1 24/57] xen/riscv: introduce asm/percpu.h
[PATCH v1 29/57] xen/riscv: introduce asm/hardirq.h
[PATCH v1 33/57] xen/riscv: introduce asm/altp2m.h
[PATCH v1 38/57] xen/riscv: introduce asm/monitor.h
[PATCH v1 39/57] xen/riscv: introduce asm/numa.h
[PATCH v1 42/57] xen/riscv: introduce asm/softirq.h
- xen/lib.h in most of the cases were changed to xen/bug.h as
mostly functionilty of bug.h is used.
- align arch-riscv.h with Arm's version of it.
- change the Author of commit with introduction of asm/atomic.h.
- update some definition from spinlock.h.
- code style changes.
---

Oleksii Kurochko (8):
xen/riscv: disable unnecessary configs
xen: introduce generic non-atomic test_*bit() xen/riscv: introduce bitops.h
xen/riscv: add definition of __read_mostly xen/riscv: add minimal stuff to mm.h to build full Xen xen/riscv: add minimal amount of stubs to build full Xen xen/riscv: enable full Xen build
xen/README: add compiler and binutils versions for RISC-V64

 README                                  |   4 +
 automation/gitlab-ci/build.yaml         |   4 +
 xen/arch/arm/include/asm/bitops.h       |  67 ----
 xen/arch/ppc/include/asm/bitops.h       |  52 ---
 xen/arch/ppc/include/asm/page.h         |   2 +-
 xen/arch/ppc/mm-radix.c                 |   2 +-
 xen/arch/riscv/Makefile                 |  17 +-
 xen/arch/riscv/arch.mk                  |   4 -
 xen/arch/riscv/configs/tiny64_defconfig |  12 +-
 xen/arch/riscv/early_printk.c           | 168 ----------
 xen/arch/riscv/include/asm/bitops.h     | 137 ++++++++
 xen/arch/riscv/include/asm/cache.h      |   2 +
 xen/arch/riscv/include/asm/domain.h     |   2 +
 xen/arch/riscv/include/asm/mm.h         | 240 ++++++++++++++
 xen/arch/riscv/include/asm/p2m.h        |   2 +
 xen/arch/riscv/mm.c                     |  52 ++-
 xen/arch/riscv/setup.c                  |  10 +-
 xen/arch/riscv/stubs.c                  | 415 ++++++++++++++++++++++++
 xen/arch/riscv/traps.c                  |  25 ++
 xen/arch/x86/include/asm/bitops.h       |  31 +-
 xen/include/xen/bitops.h                | 146 +++++++++
 21 files changed, 1069 insertions(+), 325 deletions(-)
 create mode 100644 xen/arch/riscv/include/asm/bitops.h
 create mode 100644 xen/arch/riscv/stubs.c
Signature · 1 line
2.45.0
#1 lore raw

PATCH v12 1/8 xen/riscv: disable unnecessary configs

Disables unnecessary configs for two cases:
1. By utilizing EXTRA_FIXED_RANDCONFIG for randconfig builds (GitLab CI jobs).
2. By using tiny64_defconfig for non-randconfig builds.

Only configs which lead to compilation issues were disabled.

Remove lines related to disablement of configs which aren't affected compilation:
-# CONFIG_SCHED_CREDIT is not set
-# CONFIG_SCHED_RTDS is not set
-# CONFIG_SCHED_NULL is not set
-# CONFIG_SCHED_ARINC653 is not set
-# CONFIG_TRACEBUFFER is not set
-# CONFIG_HYPFS is not set
-# CONFIG_SPECULATIVE_HARDEN_ARRAY is not set

To allow CONFIG_ARGO build happy it was included <asm/p2m.h> to <asm/domain.h> as ARGO requires p2m_type_t ( p2m_ram_rw ) and declaration of check_get_page_from_gfn() from xen/p2m-common.h.

Also, it was included <xen/errno.h> to asm/p2m.h as after the latter was included to <asm/domain.h> the compilation error that EINVAL, EOPNOTSUPP aren't declared started to occur.

CONFIG_XSM=n as it requires an introduction of:
* boot_module_find_by_kind()
* BOOTMOD_XSM
* struct bootmodule
* copy_from_paddr()
The mentioned things aren't introduced now.

CPU_BOOT_TIME_CPUPOOLS requires an introduction of cpu_physical_id() and acpi_disabled, so it is disabled for now.

PERF_COUNTERS requires asm/perf.h and asm/perfc-defn.h, so it is also disabled for now, as RISC-V hasn't introduced this headers yet.

LIVEPATCH isn't ready for RISC-V too and it can be overriden by randconfig, so to avoid compilation errors for randconfig it is disabled for now.

---
Changes in V10-V12:
- Nothing changed. Only rebase.
---
Changes in V9:
- update the commit message: add info about LIVEPATCH and PERF_COUNTERS.
---
Changes in V8:
- disabled CPU_BOOT_TIME_CPUPOOLS as it requires an introduction of cpu_physical_id() and acpi_disabled.
- leave XSM disabled, add explanation in the commit message.
- drop HYPFS as the patch was provided to resolve compilation issue when this condif is enabled for RISC-V.
- include asm/p2m.h to asm/domain.h, and xen/errno.h to asm/p2m.h to drop ARGO config from tiny64_defconfing and build.yaml.
- update the commit message.
---
Changes in V7:
- Disable only configs which cause compilation issues.
- Update the commit message.
---
Changes in V6:
- Nothing changed. Only rebase.
---
Changes in V5:
- Rebase and drop duplicated configs in EXTRA_FIXED_RANDCONFIG list
- Update the commit message
---
Changes in V4:
- Nothing changed. Only rebase
---
Changes in V3:
- Remove EXTRA_FIXED_RANDCONFIG for non-randconfig jobs.
For non-randconfig jobs, it is sufficient to disable configs by using the defconfig.
- Remove double blank lines in build.yaml file before archlinux-current-gcc-riscv64-debug
---
Changes in V2:
- update the commit message.
- remove xen/arch/riscv/Kconfig changes.

---
 automation/gitlab-ci/build.yaml         |  4 ++++
 xen/arch/riscv/configs/tiny64_defconfig | 12 +++++-------
 xen/arch/riscv/include/asm/domain.h     |  2 ++
 xen/arch/riscv/include/asm/p2m.h        |  2 ++
 4 files changed, 13 insertions(+), 7 deletions(-)
automation/gitlab-ci/build.yaml +4 -0
index 5985be9378..3290a36dca 100644
@@ -379,10 +379,14 @@ alpine-3.18-gcc-debug-arm64:
 .riscv-fixed-randconfig:
   variables: &riscv-fixed-randconfig
     EXTRA_FIXED_RANDCONFIG: |
+      CONFIG_BOOT_TIME_CPUPOOLS=n
       CONFIG_COVERAGE=n
       CONFIG_EXPERT=y
       CONFIG_GRANT_TABLE=n
       CONFIG_MEM_ACCESS=n
+      CONFIG_PERF_COUNTERS=n
+      CONFIG_LIVEPATCH=n
+      CONFIG_XSM=n
 
 archlinux-current-gcc-riscv64-debug:
   extends: .gcc-riscv64-cross-build-debug
xen/arch/riscv/configs/tiny64_defconfig +5 -7
index 09defe236b..fc7a04872f 100644
@@ -1,12 +1,10 @@
-# CONFIG_SCHED_CREDIT is not set
-# CONFIG_SCHED_RTDS is not set
-# CONFIG_SCHED_NULL is not set
-# CONFIG_SCHED_ARINC653 is not set
-# CONFIG_TRACEBUFFER is not set
-# CONFIG_HYPFS is not set
+# CONFIG_BOOT_TIME_CPUPOOLS is not set
 # CONFIG_GRANT_TABLE is not set
-# CONFIG_SPECULATIVE_HARDEN_ARRAY is not set
 # CONFIG_MEM_ACCESS is not set
+# CONFIG_PERF_COUNTERS is not set
+# CONFIG_COVERAGE is not set
+# CONFIG_LIVEPATCH is not set
+# CONFIG_XSM is not set
 
 CONFIG_RISCV_64=y
 CONFIG_DEBUG=y
xen/arch/riscv/include/asm/domain.h +2 -0
index 027bfa8a93..16a9dd57aa 100644
@@ -5,6 +5,8 @@
 #include <xen/xmalloc.h>
 #include <public/hvm/params.h>
 
+#include <asm/p2m.h>
+
 struct hvm_domain
 {
     uint64_t              params[HVM_NR_PARAMS];
xen/arch/riscv/include/asm/p2m.h +2 -0
index 387f372b5d..26860c0ae7 100644
@@ -2,6 +2,8 @@
 #ifndef __ASM_RISCV_P2M_H__
 #define __ASM_RISCV_P2M_H__
 
+#include <xen/errno.h>
+
 #include <asm/page-bits.h>
 
 #define paddr_bits PADDR_BITS
Signature · 1 line
2.45.0
Acked-by on v12 1/8
#2 lore raw

Re: [PATCH v12 1/8] xen/riscv: disable unnecessary configs

The subject should say "update Kconfig", because you're not (only) disabling.

I'd suggest "xen/riscv: Update Kconfig in preparation for a full Xen build".

Quoted 31 lines · Oleksii Kurochko

Disables unnecessary configs for two cases:
1. By utilizing EXTRA_FIXED_RANDCONFIG for randconfig builds (GitLab CI jobs).
2. By using tiny64_defconfig for non-randconfig builds.

Only configs which lead to compilation issues were disabled.

Remove lines related to disablement of configs which aren't affected compilation:
-# CONFIG_SCHED_CREDIT is not set
-# CONFIG_SCHED_RTDS is not set
-# CONFIG_SCHED_NULL is not set
-# CONFIG_SCHED_ARINC653 is not set
-# CONFIG_TRACEBUFFER is not set
-# CONFIG_HYPFS is not set
-# CONFIG_SPECULATIVE_HARDEN_ARRAY is not set

To allow CONFIG_ARGO build happy it was included <asm/p2m.h> to <asm/domain.h> as ARGO requires p2m_type_t ( p2m_ram_rw ) and declaration of check_get_page_from_gfn() from xen/p2m-common.h.

Also, it was included <xen/errno.h> to asm/p2m.h as after the latter was included to <asm/domain.h> the compilation error that EINVAL, EOPNOTSUPP aren't declared started to occur.

CONFIG_XSM=n as it requires an introduction of:
* boot_module_find_by_kind()
* BOOTMOD_XSM
* struct bootmodule
* copy_from_paddr()
The mentioned things aren't introduced now.

Quoted 2 lines

CPU_BOOT_TIME_CPUPOOLS requires an introduction of cpu_physical_id() and acpi_disabled, so it is disabled for now.

CONFIG_BOOT_TIME_CPUPOOLS

Also the "depends on DT" isn't good enough as a restriction IMO.  It's very ARM-dom0less specific.

Quoted 4 lines

PERF_COUNTERS requires asm/perf.h and asm/perfc-defn.h, so it is also disabled for now, as RISC-V hasn't introduced this headers yet. LIVEPATCH isn't ready for RISC-V too and it can be overriden by randconfig, so to avoid compilation errors for randconfig it is disabled for now.

PERF_COUNTERS is x86-only, and both LIVEPATCH really should be guarded by have HAVE_$FOO selected by ARCH.

However, that's not work to get stuck into now.

It's quite unreasonable how much stuff doesn't work in simple builds...

Quoted 1 line

I'm happy to fix up the two minor issues on commit.

author
#3 lore raw

Re: [PATCH v12 1/8] xen/riscv: disable unnecessary configs

Quoted 68 lines · Andrew Cooper

The subject should say "update Kconfig", because you're not (only) disabling.

I'd suggest "xen/riscv: Update Kconfig in preparation for a full Xen build".

Quoted 36 lines · Oleksii Kurochko

Disables unnecessary configs for two cases:
1. By utilizing EXTRA_FIXED_RANDCONFIG for randconfig builds (GitLab CI jobs).
2. By using tiny64_defconfig for non-randconfig builds.

Only configs which lead to compilation issues were disabled.

Remove lines related to disablement of configs which aren't affected
compilation:
 -# CONFIG_SCHED_CREDIT is not set
 -# CONFIG_SCHED_RTDS is not set
 -# CONFIG_SCHED_NULL is not set
 -# CONFIG_SCHED_ARINC653 is not set
 -# CONFIG_TRACEBUFFER is not set
 -# CONFIG_HYPFS is not set
 -# CONFIG_SPECULATIVE_HARDEN_ARRAY is not set

To allow CONFIG_ARGO build happy it was included <asm/p2m.h> to <asm/domain.h>
as ARGO requires p2m_type_t ( p2m_ram_rw ) and declaration of check_get_page_from_gfn() from xen/p2m-common.h.

Also, it was included <xen/errno.h> to asm/p2m.h as after the latter was
included to <asm/domain.h> the compilation error that EINVAL, EOPNOTSUPP
aren't declared started to occur.

CONFIG_XSM=n as it requires an introduction of:
* boot_module_find_by_kind()
* BOOTMOD_XSM
* struct bootmodule
* copy_from_paddr()
The mentioned things aren't introduced now.

Quoted 3 lines

CPU_BOOT_TIME_CPUPOOLS requires an introduction of cpu_physical_id() and
acpi_disabled, so it is disabled for now.

CONFIG_BOOT_TIME_CPUPOOLS

Also the "depends on DT" isn't good enough as a restriction IMO. It's
very ARM-dom0less specific.

Quoted 7 lines

PERF_COUNTERS requires asm/perf.h and asm/perfc-defn.h, so it is also disabled for now, as RISC-V hasn't introduced this headers yet.
LIVEPATCH isn't ready for RISC-V too and it can be overriden by randconfig,
so to avoid compilation errors for randconfig it is disabled for now.

PERF_COUNTERS is x86-only, and both LIVEPATCH really should be guarded
by have HAVE_$FOO selected by ARCH.

However, that's not work to get stuck into now.

Quoted 8 lines

It's quite unreasonable how much stuff doesn't work in simple builds...

Quoted 1 line

I'm happy to fix up the two minor issues on commit.

Thanks. I Would appreciate that.

~ Oleksii

#4 lore raw

Re: [PATCH v12 1/8] xen/riscv: disable unnecessary configs

In reply to Andrew Cooper
Quoted 4 lines · Andrew Cooper
Quoted 3 lines · Oleksii Kurochko

To allow CONFIG_ARGO build happy it was included <asm/p2m.h> to <asm/domain.h> as ARGO requires p2m_type_t ( p2m_ram_rw ) and declaration of check_get_page_from_gfn() from xen/p2m-common.h.

Actually, this is an ARGO bug not a RISC-V bug.

It only builds in other architectures because of transitive dependencies through asm/domain.h

I've fixed up argo.c to include the right header, which also prevents (for now at least) adding to general tangle which is asm/domain.h

~Andrew

#5 lore raw

PATCH v12 2/8 xen: introduce generic non-atomic test_*bit()

In reply to Oleksii Kurochko

The following generic functions were introduced:
* test_bit
* generic__test_and_set_bit
* generic__test_and_clear_bit
* generic__test_and_change_bit

These functions and macros can be useful for architectures that don't have corresponding arch-specific instructions.

Also, the patch introduces the following generics which are used by the functions mentioned above:
* BITOP_BITS_PER_WORD
* BITOP_MASK
* BITOP_WORD
* BITOP_TYPE

The following approach was chosen for generic*() and arch*() bit operation functions:
If the bit operation function that is going to be generic starts with the prefix "__", then the corresponding generic/arch function will also contain the "__" prefix. For example:
* test_bit() will be defined using arch_test_bit() and generic_test_bit().
* __test_and_set_bit() will be defined using arch__test_and_set_bit() and generic__test_and_set_bit().

---
Changes in V12:
- revert change of moving the definition of BITS_PER_BYTE from <arch>/bitops.h to xen/bitops.h.
( a separate patch will be provided to put BITS_PER_BYTE to proper place )
- drop comments on top of generic_*() functions and update the comments above __test_*() and test_bit().
- update how static inline __test_*() are defined ( drop pointless fallback #define ) and test_bit().
- drop the footer after Signed-off-by.
---
Changes in V11:
- fix identation in generic_test_bit() function.
- move definition of BITS_PER_BYTE from <arch>/bitops.h to xen/bitops.h
- drop the changes in arm64/livepatch.c.
- update the the comments on top of functions: generic__test_and_set_bit(), generic__test_and_clear_bit(), generic__test_and_change_bit(), generic_test_bit().
- Update footer after Signed-off section.
- Rebase the patch on top of staging branch, so it can be merged when necessary approves will be given.
- ? add Reviewed-by: Jan Beulich <[email protected]>.
---
Changes in V10:
- update the commit message. ( re-order paragraphs and add explanation usage of prefix "__" in bit
operation function names )
- add parentheses around the whole expression of bitop_bad_size() macros.
- move macros bitop_bad_size() above asm/bitops.h as it is not arch-specific anymore and there is
no need for overriding it.
- drop macros check_bitop_size() and use "if ( bitop_bad_size(addr) ) __bitop_bad_size();" implictly
where it is needed.
- in <xen/bitops.h> use 'int' as a first parameter for __test_and_*(), generic__test_and_*() to be
consistent with how the mentioned functions were declared in the original per-arch functions.
- add 'const' to p variable in generic_test_bit().
- move definition of BITOP_BITS_PER_WORD and bitop_uint_t to xen/bitops.h as we don't allow for arch
overrides these definitions anymore.
---
Changes in V9:
- move up xen/bitops.h in ppc/asm/page.h.
- update defintion of arch_check_bitop_size.
And drop correspondent macros from x86/asm/bitops.h
- drop parentheses in generic__test_and_set_bit() for definition of
local variable p.
- fix indentation inside #ifndef BITOP_TYPE...#endif
- update the commit message.
---
Changes in V8:
- drop __pure for function which uses volatile.
- drop unnessary () in generic__test_and_change_bit() for addr casting.
- update prototype of generic_test_bit() and test_bit(): now it returns bool
instead of int.
- update generic_test_bit() to use BITOP_MASK().
- Deal with fls{l} changes: it should be in the patch with introduced generic fls{l}.
- add a footer with explanation of dependency on an uncommitted patch after Signed-off.
- abstract bitop_size().
- move BITOP_TYPE define to <xen/types.h>.
---
Changes in V7:
- move everything to xen/bitops.h to follow the same approach for all generic
bit ops.
- put together BITOP_BITS_PER_WORD and bitops_uint_t.
- make BITOP_MASK more generic.
- drop #ifdef ... #endif around BITOP_MASK, BITOP_WORD as they are generic
enough.
- drop "_" for generic__{test_and_set_bit,...}().
- drop " != 0" for functions which return bool.
- add volatile during the cast for generic__{...}().
- update the commit message.
- update arch related code to follow the proposed generic approach.
---
Changes in V6:
- Nothing changed ( only rebase )
---
Changes in V5:
- new patch

---
 xen/arch/arm/include/asm/bitops.h |  67 --------------
 xen/arch/ppc/include/asm/bitops.h |  52 -----------
 xen/arch/ppc/include/asm/page.h   |   2 +-
 xen/arch/ppc/mm-radix.c           |   2 +-
 xen/arch/x86/include/asm/bitops.h |  31 ++-----
 xen/include/xen/bitops.h          | 146 ++++++++++++++++++++++++++++++
 6 files changed, 157 insertions(+), 143 deletions(-)
xen/arch/arm/include/asm/bitops.h +0 -67
index 8f4bdc09d1..3c023103f7 100644
@@ -22,9 +22,6 @@
 #define __set_bit(n,p)            set_bit(n,p)
 #define __clear_bit(n,p)          clear_bit(n,p)
 
-#define BITOP_BITS_PER_WORD     32
-#define BITOP_MASK(nr)          (1UL << ((nr) % BITOP_BITS_PER_WORD))
-#define BITOP_WORD(nr)          ((nr) / BITOP_BITS_PER_WORD)
 #define BITS_PER_BYTE           8
 
 #define ADDR (*(volatile int *) addr)
@@ -76,70 +73,6 @@ bool test_and_change_bit_timeout(int nr, volatile void *p,
 bool clear_mask16_timeout(uint16_t mask, volatile void *p,
                           unsigned int max_try);
 
-/**
- * __test_and_set_bit - Set a bit and return its old value
- * @nr: Bit to set
- * @addr: Address to count from
- *
- * This operation is non-atomic and can be reordered.
- * If two examples of this operation race, one can appear to succeed
- * but actually fail.  You must protect multiple accesses with a lock.
- */
-static inline int __test_and_set_bit(int nr, volatile void *addr)
-{
-        unsigned int mask = BITOP_MASK(nr);
-        volatile unsigned int *p =
-                ((volatile unsigned int *)addr) + BITOP_WORD(nr);
-        unsigned int old = *p;
-
-        *p = old | mask;
-        return (old & mask) != 0;
-}
-
-/**
- * __test_and_clear_bit - Clear a bit and return its old value
- * @nr: Bit to clear
- * @addr: Address to count from
- *
- * This operation is non-atomic and can be reordered.
- * If two examples of this operation race, one can appear to succeed
- * but actually fail.  You must protect multiple accesses with a lock.
- */
-static inline int __test_and_clear_bit(int nr, volatile void *addr)
-{
-        unsigned int mask = BITOP_MASK(nr);
-        volatile unsigned int *p =
-                ((volatile unsigned int *)addr) + BITOP_WORD(nr);
-        unsigned int old = *p;
-
-        *p = old & ~mask;
-        return (old & mask) != 0;
-}
-
-/* WARNING: non atomic and it can be reordered! */
-static inline int __test_and_change_bit(int nr,
-                                            volatile void *addr)
-{
-        unsigned int mask = BITOP_MASK(nr);
-        volatile unsigned int *p =
-                ((volatile unsigned int *)addr) + BITOP_WORD(nr);
-        unsigned int old = *p;
-
-        *p = old ^ mask;
-        return (old & mask) != 0;
-}
-
-/**
- * test_bit - Determine whether a bit is set
- * @nr: bit number to test
- * @addr: Address to start counting from
- */
-static inline int test_bit(int nr, const volatile void *addr)
-{
-        const volatile unsigned int *p = (const volatile unsigned int *)addr;
-        return 1UL & (p[BITOP_WORD(nr)] >> (nr & (BITOP_BITS_PER_WORD-1)));
-}
-
 #define arch_ffs(x)  ((x) ? 1 + __builtin_ctz(x) : 0)
 #define arch_ffsl(x) ((x) ? 1 + __builtin_ctzl(x) : 0)
 #define arch_fls(x)  ((x) ? 32 - __builtin_clz(x) : 0)
xen/arch/ppc/include/asm/bitops.h +0 -52

70 lines not on this page · raw message

xen/arch/ppc/include/asm/page.h +1 -1
index 890e285051..6d4cd2611c 100644
@@ -2,9 +2,9 @@
 #ifndef _ASM_PPC_PAGE_H
 #define _ASM_PPC_PAGE_H
 
+#include <xen/bitops.h>
 #include <xen/types.h>
 
-#include <asm/bitops.h>
 #include <asm/byteorder.h>
 
 #define PDE_VALID     PPC_BIT(0)
xen/arch/ppc/mm-radix.c +1 -1
index ab5a10695c..9055730997 100644
@@ -1,11 +1,11 @@
 /* SPDX-License-Identifier: GPL-2.0-or-later */
+#include <xen/bitops.h>
 #include <xen/init.h>
 #include <xen/kernel.h>
 #include <xen/mm.h>
 #include <xen/types.h>
 #include <xen/lib.h>
 
-#include <asm/bitops.h>
 #include <asm/byteorder.h>
 #include <asm/early_printk.h>
 #include <asm/page.h>
xen/arch/x86/include/asm/bitops.h +9 -22
index fc9fe73ad5..8ee5cbf127 100644
@@ -19,9 +19,6 @@
 #define ADDR (*(volatile int *) addr)
 #define CONST_ADDR (*(const volatile int *) addr)
 
-extern void __bitop_bad_size(void);
-#define bitop_bad_size(addr) (sizeof(*(addr)) < 4)
-
 /**
  * set_bit - Atomically set a bit in memory
  * @nr: the bit to set
@@ -175,7 +172,7 @@ static inline int test_and_set_bit(int nr, volatile void *addr)
 })
 
 /**
- * __test_and_set_bit - Set a bit and return its old value
+ * arch__test_and_set_bit - Set a bit and return its old value
  * @nr: Bit to set
  * @addr: Address to count from
  *
@@ -183,7 +180,7 @@ static inline int test_and_set_bit(int nr, volatile void *addr)
  * If two examples of this operation race, one can appear to succeed
  * but actually fail.  You must protect multiple accesses with a lock.
  */
-static inline int __test_and_set_bit(int nr, void *addr)
+static inline int arch__test_and_set_bit(int nr, volatile void *addr)
 {
     int oldbit;
 
@@ -194,10 +191,7 @@ static inline int __test_and_set_bit(int nr, void *addr)
 
     return oldbit;
 }
-#define __test_and_set_bit(nr, addr) ({                 \
-    if ( bitop_bad_size(addr) ) __bitop_bad_size();     \
-    __test_and_set_bit(nr, addr);                       \
-})
+#define arch__test_and_set_bit arch__test_and_set_bit
 
 /**
  * test_and_clear_bit - Clear a bit and return its old value
@@ -224,7 +218,7 @@ static inline int test_and_clear_bit(int nr, volatile void *addr)
 })
 
 /**
- * __test_and_clear_bit - Clear a bit and return its old value
+ * arch__test_and_clear_bit - Clear a bit and return its old value
  * @nr: Bit to set
  * @addr: Address to count from
  *
@@ -232,7 +226,7 @@ static inline int test_and_clear_bit(int nr, volatile void *addr)
  * If two examples of this operation race, one can appear to succeed
  * but actually fail.  You must protect multiple accesses with a lock.
  */
-static inline int __test_and_clear_bit(int nr, void *addr)
+static inline int arch__test_and_clear_bit(int nr, volatile void *addr)
 {
     int oldbit;
 
@@ -243,13 +237,10 @@ static inline int __test_and_clear_bit(int nr, void *addr)
 
     return oldbit;
 }
-#define __test_and_clear_bit(nr, addr) ({               \
-    if ( bitop_bad_size(addr) ) __bitop_bad_size();     \
-    __test_and_clear_bit(nr, addr);                     \
-})
+#define arch__test_and_clear_bit arch__test_and_clear_bit
 
 /* WARNING: non atomic and it can be reordered! */
-static inline int __test_and_change_bit(int nr, void *addr)
+static inline int arch__test_and_change_bit(int nr, volatile void *addr)
 {
     int oldbit;
 
@@ -260,10 +251,7 @@ static inline int __test_and_change_bit(int nr, void *addr)
 
     return oldbit;
 }
-#define __test_and_change_bit(nr, addr) ({              \
-    if ( bitop_bad_size(addr) ) __bitop_bad_size();     \
-    __test_and_change_bit(nr, addr);                    \
-})
+#define arch__test_and_change_bit arch__test_and_change_bit
 
 /**
  * test_and_change_bit - Change a bit and return its new value
@@ -307,8 +295,7 @@ static inline int variable_test_bit(int nr, const volatile void *addr)
     return oldbit;
 }
 
-#define test_bit(nr, addr) ({                           \
-    if ( bitop_bad_size(addr) ) __bitop_bad_size();     \
+#define arch_test_bit(nr, addr) ({                      \
     __builtin_constant_p(nr) ?                          \
         constant_test_bit(nr, addr) :                   \
         variable_test_bit(nr, addr);                    \
xen/include/xen/bitops.h +146 -0

158 lines not on this page · raw message

Signature · 1 line
2.45.0
Reviewed-by on v12 2/8
#6 lore raw

Re: [PATCH v12 2/8] xen: introduce generic non-atomic test_*bit()

Quoted 26 lines · Oleksii Kurochko

The following generic functions were introduced:
* test_bit
* generic__test_and_set_bit
* generic__test_and_clear_bit
* generic__test_and_change_bit

These functions and macros can be useful for architectures that don't have corresponding arch-specific instructions.

Also, the patch introduces the following generics which are used by the functions mentioned above:
* BITOP_BITS_PER_WORD
* BITOP_MASK
* BITOP_WORD
* BITOP_TYPE

The following approach was chosen for generic*() and arch*() bit operation functions:
If the bit operation function that is going to be generic starts with the prefix "__", then the corresponding generic/arch function will also contain the "__" prefix. For example:
* test_bit() will be defined using arch_test_bit() and generic_test_bit().
* __test_and_set_bit() will be defined using arch__test_and_set_bit() and generic__test_and_set_bit().

Quoted 1 line

Albeit I think you cound have gone further ...

xen/arch/x86/include/asm/bitops.h static inline int variable_test_bit(int nr, const volatile void *addr) · line 301

Quoted 10 lines
xen/arch/x86/include/asm/bitops.h +1 -2
@@ -307,8 +295,7 @@ static inline int variable_test_bit(int nr, const volatile void *addr)
     return oldbit;
 }
 
-#define test_bit(nr, addr) ({                           \
-    if ( bitop_bad_size(addr) ) __bitop_bad_size();     \
+#define arch_test_bit(nr, addr) ({                      \
     __builtin_constant_p(nr) ?                          \
         constant_test_bit(nr, addr) :                   \
         variable_test_bit(nr, addr);                    \

... here, as constant_test_bit() is functionally the same as generic_test_bit(), afaict. But that can well be cleaned up subsequently, in order to no further delay this work of yours.

Jan

#7 lore raw

PATCH v12 3/8 xen/riscv: introduce bitops.h

In reply to Oleksii Kurochko

Taken from Linux-6.4.0-rc1

Xen's bitops.h consists of several Linux's headers:
* linux/arch/include/asm/bitops.h:
* The following function were removed as they aren't used in Xen:
* test_and_set_bit_lock
* clear_bit_unlock
* __clear_bit_unlock
* The following functions were renamed in the way how they are used by common code:
* __test_and_set_bit
* __test_and_clear_bit
* The declaration and implementation of the following functios were updated to make Xen build happy:
* clear_bit
* set_bit
* __test_and_clear_bit
* __test_and_set_bit

---
Changes in V11-V12:
- Nothing changed. Only rebase was done.
---
Changes in V10:
- update the error message BITS_PER_LONG -> BITOP_BITS_PER_WORD
---
Changes in V9:
- add Acked-by: Jan Beulich <[email protected]>
- drop redefinition of bitop_uint_t in asm/types.h as some operation in Xen common code expects to work with 32-bit quantities.
- s/BITS_PER_LONG/BITOP_BITS_PER_WORD in asm/bitops.h around __AMO() macros.
---
Changes in V8:
- define bitop_uint_t in <asm/types.h> after the changes in patch related to introduction of "introduce generic non-atomic test_*bit()".
- drop duplicated __set_bit() and __clear_bit().
- drop duplicated comment: /* Based on linux/arch/include/asm/bitops.h */.
- update type of res and mask in test_and_op_bit_ord(): unsigned long -> bitop_uint_t.
- drop 1 padding blank in test_and_op_bit_ord().
- update definition of test_and_set_bit(),test_and_clear_bit(),test_and_change_bit:
change return type to bool.
- change addr argument type of test_and_change_bit(): unsigned long * -> void *.
- move test_and_change_bit() closer to other test_and-s function.
- Code style fixes: tabs -> space.
- s/#undef __op_bit/#undef op_bit.
- update the commit message: delete information about generic-non-atomic.h changes as now it is a separate patch.
---
Changes in V7:
- Update the commit message.
- Drop "__" for __op_bit and __op_bit_ord as they are atomic.
- add comment above __set_bit and __clear_bit about why they are defined as atomic.
- align bitops_uint_t with __AMO().
- make changes after generic non-atomic test_*bit() were changed.
- s/__asm__ __volatile__/asm volatile
---
Changes in V6:
- rebase clean ups were done: drop unused asm-generic includes
---
Changes in V5:
- new patch

---
 xen/arch/riscv/include/asm/bitops.h | 137 ++++++++++++++++++++++++++++
 1 file changed, 137 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/bitops.h
xen/arch/riscv/include/asm/bitops.h new file +137 -0
new file mode 100644
index 0000000000..7f7af3fda1
@@ -0,0 +1,137 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2012 Regents of the University of California */
+
+#ifndef _ASM_RISCV_BITOPS_H
+#define _ASM_RISCV_BITOPS_H
+
+#include <asm/system.h>
+
+#if BITOP_BITS_PER_WORD == 64
+#define __AMO(op)   "amo" #op ".d"
+#elif BITOP_BITS_PER_WORD == 32
+#define __AMO(op)   "amo" #op ".w"
+#else
+#error "Unexpected BITOP_BITS_PER_WORD"
+#endif
+
+/* Based on linux/arch/include/asm/bitops.h */
+
+/*
+ * Non-atomic bit manipulation.
+ *
+ * Implemented using atomics to be interrupt safe. Could alternatively
+ * implement with local interrupt masking.
+ */
+#define __set_bit(n, p)      set_bit(n, p)
+#define __clear_bit(n, p)    clear_bit(n, p)
+
+#define test_and_op_bit_ord(op, mod, nr, addr, ord)     \
+({                                                      \
+    bitop_uint_t res, mask;                             \
+    mask = BITOP_MASK(nr);                              \
+    asm volatile (                                      \
+        __AMO(op) #ord " %0, %2, %1"                    \
+        : "=r" (res), "+A" (addr[BITOP_WORD(nr)])       \
+        : "r" (mod(mask))                               \
+        : "memory");                                    \
+    ((res & mask) != 0);                                \
+})
+
+#define op_bit_ord(op, mod, nr, addr, ord)      \
+    asm volatile (                              \
+        __AMO(op) #ord " zero, %1, %0"          \
+        : "+A" (addr[BITOP_WORD(nr)])           \
+        : "r" (mod(BITOP_MASK(nr)))             \
+        : "memory");
+
+#define test_and_op_bit(op, mod, nr, addr)    \
+    test_and_op_bit_ord(op, mod, nr, addr, .aqrl)
+#define op_bit(op, mod, nr, addr) \
+    op_bit_ord(op, mod, nr, addr, )
+
+/* Bitmask modifiers */
+#define NOP(x)    (x)
+#define NOT(x)    (~(x))
+
+/**
+ * test_and_set_bit - Set a bit and return its old value
+ * @nr: Bit to set
+ * @addr: Address to count from
+ */
+static inline bool test_and_set_bit(int nr, volatile void *p)
+{
+    volatile bitop_uint_t *addr = p;
+
+    return test_and_op_bit(or, NOP, nr, addr);
+}
+
+/**
+ * test_and_clear_bit - Clear a bit and return its old value
+ * @nr: Bit to clear
+ * @addr: Address to count from
+ */
+static inline bool test_and_clear_bit(int nr, volatile void *p)
+{
+    volatile bitop_uint_t *addr = p;
+
+    return test_and_op_bit(and, NOT, nr, addr);
+}
+
+/**
+ * test_and_change_bit - Toggle (change) a bit and return its old value
+ * @nr: Bit to change
+ * @addr: Address to count from
+ *
+ * This operation is atomic and cannot be reordered.
+ * It also implies a memory barrier.
+ */
+static inline bool test_and_change_bit(int nr, volatile void *p)
+{
+    volatile bitop_uint_t *addr = p;
+
+    return test_and_op_bit(xor, NOP, nr, addr);
+}
+
+/**
+ * set_bit - Atomically set a bit in memory
+ * @nr: the bit to set
+ * @addr: the address to start counting from
+ *
+ * Note that @nr may be almost arbitrarily large; this function is not
+ * restricted to acting on a single-word quantity.
+ */
+static inline void set_bit(int nr, volatile void *p)
+{
+    volatile bitop_uint_t *addr = p;
+
+    op_bit(or, NOP, nr, addr);
+}
+
+/**
+ * clear_bit - Clears a bit in memory
+ * @nr: Bit to clear
+ * @addr: Address to start counting from
+ */
+static inline void clear_bit(int nr, volatile void *p)
+{
+    volatile bitop_uint_t *addr = p;
+
+    op_bit(and, NOT, nr, addr);
+}
+
+#undef test_and_op_bit
+#undef op_bit
+#undef NOP
+#undef NOT
+#undef __AMO
+
+#endif /* _ASM_RISCV_BITOPS_H */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
Signature · 1 line
2.45.0
#8 lore raw

PATCH v12 4/8 xen/riscv: add definition of __read_mostly

In reply to Oleksii Kurochko

The definition of __read_mostly should be removed in:
https://lore.kernel.org/xen-devel/[email protected]/

The patch introduces it in arch-specific header to not block enabling of full Xen build for RISC-V.

---
- [PATCH] move __read_mostly to xen/cache.h [2]

Right now, the patch series doesn't have a direct dependency on [2] and it provides __read_mostly in the patch:
[PATCH v3 26/34] xen/riscv: add definition of __read_mostly
However, it will be dropped as soon as [2] is merged or at least when the final version of the patch [2] is provided.

Considering that there is still no still final decision regarding patch [2] my suggestion is to merge RISC-V specific patch and just drop the changes in patch [2].

[2] https://lore.kernel.org/xen-devel/[email protected]/
---
Changes in V9-V12:
- Only rebase was done.
---
Change in V8:
- update the footer after Signed-off.
---
Changes in V4-V7:
- Nothing changed. Only rebase.

---
 xen/arch/riscv/include/asm/cache.h | 2 ++
 1 file changed, 2 insertions(+)
xen/arch/riscv/include/asm/cache.h +2 -0
index 69573eb051..94bd94db53 100644
@@ -3,4 +3,6 @@
 #ifndef _ASM_RISCV_CACHE_H
 #define _ASM_RISCV_CACHE_H
 
+#define __read_mostly __section(".data.read_mostly")
+
 #endif /* _ASM_RISCV_CACHE_H */
Signature · 1 line
2.45.0
#9 lore raw

PATCH v12 5/8 xen/riscv: add minimal stuff to mm.h to build full Xen

In reply to Oleksii Kurochko

---
Changes in V8-V12:
- Nothing changed only rebase.
---
Changes in V7:
- update argument type of maddr_to_virt() function: unsigned long -> paddr_t
- rename argument of PFN_ORDER(): pfn -> pg.
- add Acked-by: Jan Beulich <[email protected]>
---
Changes in V6:
- drop __virt_to_maddr() ( transform to macro ) and __maddr_to_virt ( rename to maddr_to_virt ).
- parenthesize va in definition of vmap_to_mfn().
- Code style fixes.
---
Changes in V5:
- update the comment around "struct domain *domain;" : zero -> NULL
- fix ident. for unsigned long val;
- put page_to_virt() and virt_to_page() close to each other.
- drop unnessary leading underscore
- drop a space before the comment: /* Count of uses of this frame as its current type. */
- drop comment about a page 'not as a shadow'. it is not necessary for RISC-V
---
Changes in V4:
- update an argument name of PFN_ORDERN macros.
- drop pad at the end of 'struct page_info'.
- Change message -> subject in "Changes in V3"
- delete duplicated macros from riscv/mm.h
- fix identation in struct page_info
- align comment for PGC_ macros
- update definitions of domain_set_alloc_bitsize() and domain_clamp_alloc_bitsize()
- drop unnessary comments.
- s/BUG/BUG_ON("...")
- define __virt_to_maddr, __maddr_to_virt as stubs
- add inclusion of xen/mm-frame.h for mfn_x and others
- include "xen/mm.h" instead of "asm/mm.h" to fix compilation issues:
In file included from arch/riscv/setup.c:7:
./arch/riscv/include/asm/mm.h:60:28: error: field 'list' has incomplete type
60 | struct page_list_entry list;
| ^~~~
./arch/riscv/include/asm/mm.h:81:43: error: 'MAX_ORDER' undeclared here (not in a function)
81 | unsigned long first_dirty:MAX_ORDER + 1;
| ^~~~~~~~~
./arch/riscv/include/asm/mm.h:81:31: error: bit-field 'first_dirty' width not an integer constant
81 | unsigned long first_dirty:MAX_ORDER + 1;
- Define __virt_to_mfn() and __mfn_to_virt() using maddr_to_mfn() and mfn_to_maddr().
---
Changes in V3:
- update the commit title
- introduce DIRECTMAP_VIRT_START.
- drop changes related pfn_to_paddr() and paddr_to_pfn as they were remvoe in
[PATCH v2 32/39] xen/riscv: add minimal stuff to asm/page.h to build full Xen
- code style fixes.
- drop get_page_nr and put_page_nr as they don't need for time being
- drop CONFIG_STATIC_MEMORY related things
- code style fixes
---
Changes in V2:
- define stub for arch_get_dma_bitsize(void)

---
 xen/arch/riscv/include/asm/mm.h | 240 ++++++++++++++++++++++++++++++++
 xen/arch/riscv/mm.c             |   2 +-
 xen/arch/riscv/setup.c          |   2 +-
 3 files changed, 242 insertions(+), 2 deletions(-)
xen/arch/riscv/include/asm/mm.h +240 -0
index 07c7a0abba..cc4a07a71c 100644
@@ -3,11 +3,246 @@
 #ifndef _ASM_RISCV_MM_H
 #define _ASM_RISCV_MM_H
 
+#include <public/xen.h>
+#include <xen/bug.h>
+#include <xen/mm-frame.h>
+#include <xen/pdx.h>
+#include <xen/types.h>
+
 #include <asm/page-bits.h>
 
 #define pfn_to_paddr(pfn) ((paddr_t)(pfn) << PAGE_SHIFT)
 #define paddr_to_pfn(pa)  ((unsigned long)((pa) >> PAGE_SHIFT))
 
+#define paddr_to_pdx(pa)    mfn_to_pdx(maddr_to_mfn(pa))
+#define gfn_to_gaddr(gfn)   pfn_to_paddr(gfn_x(gfn))
+#define gaddr_to_gfn(ga)    _gfn(paddr_to_pfn(ga))
+#define mfn_to_maddr(mfn)   pfn_to_paddr(mfn_x(mfn))
+#define maddr_to_mfn(ma)    _mfn(paddr_to_pfn(ma))
+#define vmap_to_mfn(va)     maddr_to_mfn(virt_to_maddr((vaddr_t)(va)))
+#define vmap_to_page(va)    mfn_to_page(vmap_to_mfn(va))
+
+static inline void *maddr_to_virt(paddr_t ma)
+{
+    BUG_ON("unimplemented");
+    return NULL;
+}
+
+#define virt_to_maddr(va) ({ BUG_ON("unimplemented"); 0; })
+
+/* Convert between Xen-heap virtual addresses and machine frame numbers. */
+#define __virt_to_mfn(va)  mfn_x(maddr_to_mfn(virt_to_maddr(va)))
+#define __mfn_to_virt(mfn) maddr_to_virt(mfn_to_maddr(_mfn(mfn)))
+
+/*
+ * We define non-underscored wrappers for above conversion functions.
+ * These are overriden in various source files while underscored version
+ * remain intact.
+ */
+#define virt_to_mfn(va)     __virt_to_mfn(va)
+#define mfn_to_virt(mfn)    __mfn_to_virt(mfn)
+
+struct page_info
+{
+    /* Each frame can be threaded onto a doubly-linked list. */
+    struct page_list_entry list;
+
+    /* Reference count and various PGC_xxx flags and fields. */
+    unsigned long count_info;
+
+    /* Context-dependent fields follow... */
+    union {
+        /* Page is in use: ((count_info & PGC_count_mask) != 0). */
+        struct {
+            /* Type reference count and various PGT_xxx flags and fields. */
+            unsigned long type_info;
+        } inuse;
+
+        /* Page is on a free list: ((count_info & PGC_count_mask) == 0). */
+        union {
+            struct {
+                /*
+                 * Index of the first *possibly* unscrubbed page in the buddy.
+                 * One more bit than maximum possible order to accommodate
+                 * INVALID_DIRTY_IDX.
+                 */
+#define INVALID_DIRTY_IDX ((1UL << (MAX_ORDER + 1)) - 1)
+                unsigned long first_dirty:MAX_ORDER + 1;
+
+                /* Do TLBs need flushing for safety before next page use? */
+                bool need_tlbflush:1;
+
+#define BUDDY_NOT_SCRUBBING    0
+#define BUDDY_SCRUBBING        1
+#define BUDDY_SCRUB_ABORT      2
+                unsigned long scrub_state:2;
+            };
+
+            unsigned long val;
+        } free;
+    } u;
+
+    union {
+        /* Page is in use */
+        struct {
+            /* Owner of this page (NULL if page is anonymous). */
+            struct domain *domain;
+        } inuse;
+
+        /* Page is on a free list. */
+        struct {
+            /* Order-size of the free chunk this page is the head of. */
+            unsigned int order;
+        } free;
+    } v;
+
+    union {
+        /*
+         * Timestamp from 'TLB clock', used to avoid extra safety flushes.
+         * Only valid for: a) free pages, and b) pages with zero type count
+         */
+        uint32_t tlbflush_timestamp;
+    };
+};
+
+#define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
+
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info structures. */
+#define mfn_to_page(mfn)                                            \
+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
+#define page_to_mfn(pg)                                             \
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)
+
+static inline void *page_to_virt(const struct page_info *pg)
+{
+    return mfn_to_virt(mfn_x(page_to_mfn(pg)));
+}
+
+/* Convert between Xen-heap virtual addresses and page-info structures. */
+static inline struct page_info *virt_to_page(const void *v)
+{
+    BUG_ON("unimplemented");
+    return NULL;
+}
+
+/*
+ * Common code requires get_page_type and put_page_type.
+ * We don't care about typecounts so we just do the minimum to make it
+ * happy.
+ */
+static inline int get_page_type(struct page_info *page, unsigned long type)
+{
+    return 1;
+}
+
+static inline void put_page_type(struct page_info *page)
+{
+}
+
+static inline void put_page_and_type(struct page_info *page)
+{
+    put_page_type(page);
+    put_page(page);
+}
+
+/*
+ * RISC-V does not have an M2P, but common code expects a handful of
+ * M2P-related defines and functions. Provide dummy versions of these.
+ */
+#define INVALID_M2P_ENTRY        (~0UL)
+#define SHARED_M2P_ENTRY         (~0UL - 1UL)
+#define SHARED_M2P(_e)           ((_e) == SHARED_M2P_ENTRY)
+
+#define set_gpfn_from_mfn(mfn, pfn) do { (void)(mfn), (void)(pfn); } while (0)
+#define mfn_to_gfn(d, mfn) ((void)(d), _gfn(mfn_x(mfn)))
+
+#define PDX_GROUP_SHIFT (PAGE_SHIFT + VPN_BITS)
+
+static inline unsigned long domain_get_maximum_gpfn(struct domain *d)
+{
+    BUG_ON("unimplemented");
+    return 0;
+}
+
+static inline long arch_memory_op(int op, XEN_GUEST_HANDLE_PARAM(void) arg)
+{
+    BUG_ON("unimplemented");
+    return 0;
+}
+
+/*
+ * On RISCV, all the RAM is currently direct mapped in Xen.
+ * Hence return always true.
+ */
+static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
+{
+    return true;
+}
+
+#define PG_shift(idx)   (BITS_PER_LONG - (idx))
+#define PG_mask(x, idx) (x ## UL << PG_shift(idx))
+
+#define PGT_none          PG_mask(0, 1)  /* no special uses of this page   */
+#define PGT_writable_page PG_mask(1, 1)  /* has writable mappings?         */
+#define PGT_type_mask     PG_mask(1, 1)  /* Bits 31 or 63.                 */
+
+/* Count of uses of this frame as its current type. */
+#define PGT_count_width   PG_shift(2)
+#define PGT_count_mask    ((1UL << PGT_count_width) - 1)
+
+/*
+ * Page needs to be scrubbed. Since this bit can only be set on a page that is
+ * free (i.e. in PGC_state_free) we can reuse PGC_allocated bit.
+ */
+#define _PGC_need_scrub   _PGC_allocated
+#define PGC_need_scrub    PGC_allocated
+
+/* Cleared when the owning guest 'frees' this page. */
+#define _PGC_allocated    PG_shift(1)
+#define PGC_allocated     PG_mask(1, 1)
+/* Page is Xen heap? */
+#define _PGC_xen_heap     PG_shift(2)
+#define PGC_xen_heap      PG_mask(1, 2)
+/* Page is broken? */
+#define _PGC_broken       PG_shift(7)
+#define PGC_broken        PG_mask(1, 7)
+/* Mutually-exclusive page states: { inuse, offlining, offlined, free }. */
+#define PGC_state         PG_mask(3, 9)
+#define PGC_state_inuse   PG_mask(0, 9)
+#define PGC_state_offlining PG_mask(1, 9)
+#define PGC_state_offlined PG_mask(2, 9)
+#define PGC_state_free    PG_mask(3, 9)
+#define page_state_is(pg, st) (((pg)->count_info&PGC_state) == PGC_state_##st)
+
+/* Count of references to this frame. */
+#define PGC_count_width   PG_shift(9)
+#define PGC_count_mask    ((1UL << PGC_count_width) - 1)
+
+#define _PGC_extra        PG_shift(10)
+#define PGC_extra         PG_mask(1, 10)
+
+#define is_xen_heap_page(page) ((page)->count_info & PGC_xen_heap)
+#define is_xen_heap_mfn(mfn) \
+    (mfn_valid(mfn) && is_xen_heap_page(mfn_to_page(mfn)))
+
+#define is_xen_fixed_mfn(mfn)                                   \
+    ((mfn_to_maddr(mfn) >= virt_to_maddr((vaddr_t)_start)) &&   \
+     (mfn_to_maddr(mfn) <= virt_to_maddr((vaddr_t)_end - 1)))
+
+#define page_get_owner(p)    (p)->v.inuse.domain
+#define page_set_owner(p, d) ((p)->v.inuse.domain = (d))
+
+/* TODO: implement */
+#define mfn_valid(mfn) ({ (void)(mfn); 0; })
+
+#define domain_set_alloc_bitsize(d) ((void)(d))
+#define domain_clamp_alloc_bitsize(d, b) ((void)(d), (b))
+
+#define PFN_ORDER(pg) ((pg)->v.free.order)
+
 extern unsigned char cpu0_boot_stack[];
 
 void setup_initial_pagetables(void);
@@ -20,4 +255,9 @@ unsigned long calc_phys_offset(void);
 
 void turn_on_mmu(unsigned long ra);
 
+static inline unsigned int arch_get_dma_bitsize(void)
+{
+    return 32; /* TODO */
+}
+
 #endif /* _ASM_RISCV_MM_H */
xen/arch/riscv/mm.c +1 -1
index 053f043a3d..fe3a43be20 100644
@@ -5,12 +5,12 @@
 #include <xen/init.h>
 #include <xen/kernel.h>
 #include <xen/macros.h>
+#include <xen/mm.h>
 #include <xen/pfn.h>
 
 #include <asm/early_printk.h>
 #include <asm/csr.h>
 #include <asm/current.h>
-#include <asm/mm.h>
 #include <asm/page.h>
 #include <asm/processor.h>
 
xen/arch/riscv/setup.c +1 -1
index 6593f601c1..98a94c4c48 100644
@@ -2,9 +2,9 @@
 
 #include <xen/compile.h>
 #include <xen/init.h>
+#include <xen/mm.h>
 
 #include <asm/early_printk.h>
-#include <asm/mm.h>
 
 /* Xen stack for bringing up the first CPU. */
 unsigned char __initdata cpu0_boot_stack[STACK_SIZE]
Signature · 1 line
2.45.0
#10 lore raw

Re: [PATCH v12 5/8] xen/riscv: add minimal stuff to mm.h to build full Xen

Quoted 2 lines · Oleksii Kurochko

This patch looks like it can go in independently?  Or does it depend on having bitops.h working in practice?

However, one very strong suggestion...

Quoted 5 lines
xen/arch/riscv/include/asm/mm.h +0 -0
index 07c7a0abba..cc4a07a71c 100644

xen/arch/riscv/include/asm/mm.h line 117

Quoted 9 lines
xen/arch/riscv/include/asm/mm.h +8 -0
@@ -3,11 +3,246 @@
<snip>
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info structures. */
+#define mfn_to_page(mfn)                                            \
+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
+#define page_to_mfn(pg)                                             \
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)

Do yourself a favour and not introduce frametable_base_pdx to begin with.

Every RISC-V board I can find has things starting from 0 in physical address space, with RAM starting immediately after.

Taking the microchip board as an example, RAM actually starts at 0x8000000, which means that having frametable_base_pdx and assuming it does get set to 0x8000 (which isn't even a certainty, given that I think you'll need struct pages covering the PLICs), then what you are trading off is:

* Saving 32k of virtual address space only (no need to even allocate memory for this range of the framtable), by
* Having an extra memory load and add/sub in every page <-> mfn conversion, which is a screaming hotpath all over Xen.

It's a terribly short-sighted tradeoff.

32k of VA space might be worth saving in a 32bit build (I personally wouldn't - especially as there's no need to share Xen's VA space with guests, given no PV guests on ARM/RISC-V), but it's absolutely not at all in an a 64bit build with TB of VA space available.

Even if we do find a board with the first interesting thing in the frametable starting sufficiently away from 0 that it might be worth considering this slide, then it should still be Kconfig-able in a similar way to PDX_COMPRESSION.

You don't want to be penalising everyone because of a theoretical/weird corner case.

~Andrew

author
#11 lore raw

Re: [PATCH v12 5/8] xen/riscv: add minimal stuff to mm.h to build full Xen

Quoted 34 lines · Andrew Cooper
Quoted 2 lines · Oleksii Kurochko

This patch looks like it can go in independently?  Or does it depend on
having bitops.h working in practice?

However, one very strong suggestion...

Quoted 11 lines
xen/arch/riscv/include/asm/mm.h +4 -0
index 07c7a0abba..cc4a07a71c 100644
@@ -3,11 +3,246 @@
<snip>
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info structures.
Quoted 8 lines

*/
+#define
mfn_to_page(mfn)                                            \
+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx)) +#define
page_to_mfn(pg)                                             \
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)

Do yourself a favour and not introduce frametable_base_pdx to begin with.

Quoted 2 lines

Every RISC-V board I can find has things starting from 0 in physical address space, with RAM starting immediately after.

I checked Linux kernel and grep there:
[ok@fedora linux-aia]$ grep -Rni "memory@" arch/riscv/boot/dts/ -- exclude "*.tmp" -I arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive-2.dtsi:33:
memory@40000000 {
arch/riscv/boot/dts/starfive/jh7100-common.dtsi:28: memory@80000000
{
arch/riscv/boot/dts/microchip/mpfs-sev-kit.dts:49: ddrc_cache:
memory@1000000000 {
arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:33: ddrc_cache_lo:
memory@80000000 {
arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:37: ddrc_cache_hi:
memory@1040000000 {
arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:34: ddrc_cache_lo:
memory@80000000 {
arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:40: ddrc_cache_hi:
memory@1000000000 {
arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:22: ddrc_cache_lo:
memory@80000000 {
arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:27: ddrc_cache_hi:
memory@1000000000 {
arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:57: ddrc_cache_lo:
memory@80000000 {
arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:63: ddrc_cache_hi:
memory@1040000000 { arch/riscv/boot/dts/thead/th1520-beaglev-ahead.dts:32: memory@0 { arch/riscv/boot/dts/thead/th1520-lichee-module-4a.dtsi:14:
memory@0 {
arch/riscv/boot/dts/sophgo/cv1800b-milkv-duo.dts:26: memory@80000000
{
arch/riscv/boot/dts/sophgo/cv1812h.dtsi:12: memory@80000000 {
arch/riscv/boot/dts/sifive/hifive-unmatched-a00.dts:26: memory@80000000 {
arch/riscv/boot/dts/sifive/hifive-unleashed-a00.dts:25: memory@80000000 {
arch/riscv/boot/dts/canaan/k210.dtsi:82: sram: memory@80000000 {

And based on that majority of supported by Linux kernel boards has RAM starting not from 0 in physical address space. Am I confusing something?

Quoted 3 lines

Taking the microchip board as an example, RAM actually starts at 0x8000000,

Today we had conversation with the guy from SiFive in xen-devel channel and he mentioned that they are using "starfive visionfive2 and sifive unleashed platforms." which based on the grep above has RAM not at 0 address.

Also, QEMU uses 0x8000000.

Quoted 19 lines

which means that having frametable_base_pdx and assuming it
does get set to 0x8000 (which isn't even a certainty, given that I think
you'll need struct pages covering the PLICs), then what you are trading
off is:

* Saving 32k of virtual address space only (no need to even allocate memory for this range of the framtable), by
* Having an extra memory load and add/sub in every page <-> mfn conversion, which is a screaming hotpath all over Xen.

It's a terribly short-sighted tradeoff.

32k of VA space might be worth saving in a 32bit build (I personally wouldn't - especially as there's no need to share Xen's VA space with guests, given no PV guests on ARM/RISC-V), but it's absolutely not at all in an a 64bit build with TB of VA space available.

Quoted 4 lines

Even if we do find a board with the first interesting thing in the frametable starting sufficiently away from 0 that it might be worth considering this slide, then it should still be Kconfig-able in a similar way to PDX_COMPRESSION.

I found your tradeoffs reasonable, but I don't understand how it will work if RAM does not start from 0, as the frametable address and RAM address are linked.
I tried to look at the PDX_COMPRESSION config and couldn't find any "slide" there. Could you please clarify this for me?
If to use this "slide" would it help to avoid the mentioned above tradeoffs?

One more question: if we decide to go without frametable_base_pdx, would it be sufficient to simply remove mentions of it from the code ( at least, for now )?

~ Oleksii

Quoted 8 lines

You don't want to be penalising everyone because of a theoretical/weird
corner case.

~Andrew

#12 lore raw

Re: [PATCH v12 5/8] xen/riscv: add minimal stuff to mm.h to build full Xen

Quoted 116 lines · Oleksii K.
Quoted 32 lines · Andrew Cooper
Quoted 2 lines · Oleksii Kurochko

This patch looks like it can go in independently?  Or does it depend on
having bitops.h working in practice?

However, one very strong suggestion...

Quoted 11 lines
xen/arch/riscv/include/asm/mm.h +4 -0
index 07c7a0abba..cc4a07a71c 100644
@@ -3,11 +3,246 @@
<snip>
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info structures.
Quoted 8 lines

*/
+#define
mfn_to_page(mfn)                                            \
+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx)) +#define
page_to_mfn(pg)                                             \
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)

Do yourself a favour and not introduce frametable_base_pdx to begin with.

Quoted 2 lines

Every RISC-V board I can find has things starting from 0 in physical address space, with RAM starting immediately after.

I checked Linux kernel and grep there:

   [ok@fedora linux-aia]$ grep -Rni "memory@" arch/riscv/boot/dts/ --
   exclude "*.tmp" -I
   arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive-2.dtsi:33:    
   memory@40000000 {
   arch/riscv/boot/dts/starfive/jh7100-common.dtsi:28:     memory@80000000
   {
   arch/riscv/boot/dts/microchip/mpfs-sev-kit.dts:49:      ddrc_cache:
   memory@1000000000 {
   arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:33:   ddrc_cache_lo:
   memory@80000000 {
   arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:37:   ddrc_cache_hi:
   memory@1040000000 {
   arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:34:      ddrc_cache_lo:
   memory@80000000 {
   arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:40:      ddrc_cache_hi:
   memory@1000000000 {
   arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:22:   ddrc_cache_lo:
   memory@80000000 {
   arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:27:   ddrc_cache_hi:
   memory@1000000000 {
   arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:57:   ddrc_cache_lo:
   memory@80000000 {
   arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:63:   ddrc_cache_hi:
   memory@1040000000 {
   arch/riscv/boot/dts/thead/th1520-beaglev-ahead.dts:32:  memory@0 {
   arch/riscv/boot/dts/thead/th1520-lichee-module-4a.dtsi:14:     
   memory@0 {
   arch/riscv/boot/dts/sophgo/cv1800b-milkv-duo.dts:26:    memory@80000000
   {
   arch/riscv/boot/dts/sophgo/cv1812h.dtsi:12:     memory@80000000 {
   arch/riscv/boot/dts/sifive/hifive-unmatched-a00.dts:26: memory@80000000
   {
   arch/riscv/boot/dts/sifive/hifive-unleashed-a00.dts:25: memory@80000000
   {
   arch/riscv/boot/dts/canaan/k210.dtsi:82:        sram: memory@80000000 {

And based on that majority of supported by Linux kernel boards has RAM starting not from 0 in physical address space. Am I confusing something?

Quoted 2 lines

Taking the microchip board as an example, RAM actually starts at 0x8000000,

Today we had conversation with the guy from SiFive in xen-devel channel and he mentioned that they are using "starfive visionfive2 and sifive unleashed platforms." which based on the grep above has RAM not at 0 address.

Also, QEMU uses 0x8000000.

Quoted 19 lines

which means that having frametable_base_pdx and assuming it
does get set to 0x8000 (which isn't even a certainty, given that I think
you'll need struct pages covering the PLICs), then what you are trading
off is:

* Saving 32k of virtual address space only (no need to even allocate memory for this range of the framtable), by
* Having an extra memory load and add/sub in every page <-> mfn conversion, which is a screaming hotpath all over Xen.

It's a terribly short-sighted tradeoff.

32k of VA space might be worth saving in a 32bit build (I personally wouldn't - especially as there's no need to share Xen's VA space with guests, given no PV guests on ARM/RISC-V), but it's absolutely not at all in an a 64bit build with TB of VA space available.

Quoted 4 lines

Even if we do find a board with the first interesting thing in the frametable starting sufficiently away from 0 that it might be worth considering this slide, then it should still be Kconfig-able in a similar way to PDX_COMPRESSION.

I found your tradeoffs reasonable, but I don't understand how it will work if RAM does not start from 0, as the frametable address and RAM address are linked.
I tried to look at the PDX_COMPRESSION config and couldn't find any "slide" there. Could you please clarify this for me?
If to use this "slide" would it help to avoid the mentioned above tradeoffs?

Quoted 3 lines

One more question: if we decide to go without frametable_base_pdx, would it be sufficient to simply remove mentions of it from the code ( at least, for now )?

There is a relationship between system/host physical addresses (what Xen called maddr/mfn), and the frametable.  The frametable has one entry per mfn.

In the most simple case, there's a 1:1 relationship.  i.e. frametable[0] = maddr(0), frametable[1] = maddr(4k) etc.  This is very simple, and very easy to calculate (page_to_mfn()/mfn_to_page()).

The frametable is one big array.  It starts at a compile-time fixed address, and needs to be long enough to cover everything interesting in memory.  Therefore it potentially takes a large amount of virtual address space.

However, only interesting maddrs need to have data in the frametable, so it's fine for the backing RAM to be sparsely allocated/mapped in the frametable virtual addresses.

For 64bit, that's really all you need, because there's always far more virtual address space than physical RAM in the system, even when you're looking at TB-scale giant servers.

For 32bit, virtual address space is a limited resource.  (Also to an extent, 64bit x86 with PV guests because we give 98% of the virtual address space to the guest kernel to use.)

There are two tricks to reduce the virtual address space used, but they both cost performance in fastpaths.

1) PDX Compression.

PDX compression makes a non-linear mfn <-> maddr mapping.  This is for a usecase where you've got multiple RAM banks which are separated by a large distance (and evenly spaced), then you can "compress" a single range of 0's out of the middle of the system/host physical address.

The cost is that all page <-> mfn conversions need to read two masks and a shift-count from variables in memory, to split/shift/recombine the address bits.

2) A slide, which is frametable_base_pdx in this context.

When there's a big gap between 0 and the start of something interesting, you could chop out that range by just subtracting base_pdx.  What qualifies as "big" is subjective, but Qemu starting at 128M certainly does not qualify as big enough to warrant frametable_base_pdx.

This is less expensive that PDX compression.  It only adds one memory read to the fastpath, but it also doesn't save as much virtual address space as PDX compression.

When virtual address space is a major constraint (32 bit builds), both of these techniques are worth doing.  But when there's no constraint on virtual address space (64 bit builds), there's no reason to use either; and the performance will definitely improve as a result.

~Andrew

author
#13 lore raw

Re: [PATCH v12 5/8] xen/riscv: add minimal stuff to mm.h to build full Xen

Quoted 221 lines · Andrew Cooper
Quoted 151 lines · Oleksii K.
Quoted 35 lines · Andrew Cooper
Quoted 2 lines · Oleksii Kurochko

This patch looks like it can go in independently?  Or does it depend
on
having bitops.h working in practice?

However, one very strong suggestion...

Quoted 12 lines
xen/arch/riscv/include/asm/mm.h +4 -0
index 07c7a0abba..cc4a07a71c 100644
@@ -3,11 +3,246 @@
<snip>
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info

structures.

Quoted 8 lines

*/
+#define
mfn_to_page(mfn)                                            \
+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx)) +#define
page_to_mfn(pg)                                             \
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)

Do yourself a favour and not introduce frametable_base_pdx to begin
with.

Quoted 3 lines

Every RISC-V board I can find has things starting from 0 in physical
address space, with RAM starting immediately after.

I checked Linux kernel and grep there:
   [ok@fedora linux-aia]$ grep -Rni "memory@" arch/riscv/boot/dts/ --
   exclude "*.tmp" -I arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive- 2.dtsi:33:
   memory@40000000 { arch/riscv/boot/dts/starfive/jh7100-common.dtsi:28:
memory@80000000
   {
   arch/riscv/boot/dts/microchip/mpfs-sev-kit.dts:49:
ddrc_cache:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:33:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:37:
ddrc_cache_hi:
   memory@1040000000 { arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:34:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:40:
ddrc_cache_hi:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:22:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:27:
ddrc_cache_hi:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:57:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:63:
ddrc_cache_hi:
   memory@1040000000 { arch/riscv/boot/dts/thead/th1520-beaglev-ahead.dts:32:  memory@0 { arch/riscv/boot/dts/thead/th1520-lichee-module-4a.dtsi:14:
   memory@0 { arch/riscv/boot/dts/sophgo/cv1800b-milkv-duo.dts:26:
memory@80000000
   {
   arch/riscv/boot/dts/sophgo/cv1812h.dtsi:12:     memory@80000000 { arch/riscv/boot/dts/sifive/hifive-unmatched-a00.dts:26:
memory@80000000
   { arch/riscv/boot/dts/sifive/hifive-unleashed-a00.dts:25:
memory@80000000
   {
   arch/riscv/boot/dts/canaan/k210.dtsi:82:        sram:
memory@80000000 {

And based on  that majority of supported by Linux kernel boards has RAM
starting not from 0 in physical address space. Am I confusing something?

Quoted 2 lines

Taking the microchip board as an example, RAM actually starts at 0x8000000,

Today we had conversation with the guy from SiFive in xen-devel channel
and he mentioned that they are using "starfive visionfive2 and sifive
unleashed platforms." which based on the grep above has RAM not at 0
address.

Also, QEMU uses 0x8000000.

Quoted 24 lines

 which means that having frametable_base_pdx and assuming it does get set to 0x8000 (which isn't even a certainty, given that I
think
you'll need struct pages covering the PLICs), then what you are trading
off is:

* Saving 32k of virtual address space only (no need to even allocate
memory for this range of the framtable), by
* Having an extra memory load and add/sub in every page <-> mfn conversion, which is a screaming hotpath all over Xen.

It's a terribly short-sighted tradeoff.

32k of VA space might be worth saving in a 32bit build (I personally
wouldn't - especially as there's no need to share Xen's VA space with
guests, given no PV guests on ARM/RISC-V), but it's absolutely not at
all in an a 64bit build with TB of VA space available.

Quoted 6 lines

Even if we do find a board with the first interesting thing in the
frametable starting sufficiently away from 0 that it might be worth
considering this slide, then it should still be Kconfig-able in a similar way to PDX_COMPRESSION.

I found your tradeoffs reasonable, but I don't understand how it will
work if RAM does not start from 0, as the frametable address and RAM
address are linked.
I tried to look at the PDX_COMPRESSION config and couldn't find any "slide" there. Could you please clarify this for me?
If to use this "slide" would it help to avoid the mentioned above tradeoffs?

Quoted 4 lines

One more question: if we decide to go without frametable_base_pdx, would it be sufficient to simply remove mentions of it from the code (
at least, for now )?

There is a relationship between system/host physical addresses (what Xen
called maddr/mfn), and the frametable.  The frametable has one entry per
mfn.

In the most simple case, there's a 1:1 relationship.  i.e. frametable[0]
= maddr(0), frametable[1] = maddr(4k) etc.  This is very simple, and very easy to calculate (page_to_mfn()/mfn_to_page()).

The frametable is one big array.  It starts at a compile-time fixed address, and needs to be long enough to cover everything interesting in
memory.  Therefore it potentially takes a large amount of virtual address space.

However, only interesting maddrs need to have data in the frametable, so
it's fine for the backing RAM to be sparsely allocated/mapped in the frametable virtual addresses.

For 64bit, that's really all you need, because there's always far more
virtual address space than physical RAM in the system, even when you're
looking at TB-scale giant servers.

For 32bit, virtual address space is a limited resource.  (Also to an extent, 64bit x86 with PV guests because we give 98% of the virtual address space to the guest kernel to use.)

There are two tricks to reduce the virtual address space used, but they
both cost performance in fastpaths.

1) PDX Compression.

PDX compression makes a non-linear mfn <-> maddr mapping.  This is for a
usecase where you've got multiple RAM banks which are separated by a large distance (and evenly spaced), then you can "compress" a single range of 0's out of the middle of the system/host physical address.

The cost is that all page <-> mfn conversions need to read two masks and
a shift-count from variables in memory, to split/shift/recombine the address bits.

2) A slide, which is frametable_base_pdx in this context.

When there's a big gap between 0 and the start of something interesting,
you could chop out that range by just subtracting base_pdx.  What qualifies as "big" is subjective, but Qemu starting at 128M certainly does not qualify as big enough to warrant frametable_base_pdx.

This is less expensive that PDX compression.  It only adds one memory read to the fastpath, but it also doesn't save as much virtual address
space as PDX compression.

Quoted 7 lines

When virtual address space is a major constraint (32 bit builds), both
of these techniques are worth doing.  But when there's no constraint on
virtual address space (64 bit builds), there's no reason to use either;
and the performance will definitely improve as a result.

Thanks for such good explanation.

For RISC-V we have PDX config disabled as I haven't seen multiple RAM banks at boards which has hypervisor extension. Thereby mfn_to_pdx() and pdx_to_mfn() do nothing. The same for frametable_base_pdx, in case of PDX disabled, it just an offset ( or a slide ).

IIUUC, you meant that it make sense to map frametable not to the address of where RAM is started, but to 0, so then we don't need this +-frametable_base_pdx. The price for that is loosing of VA space for the range from 0 to RAM start address.

Right now, we are trying to support 3 boards with the following RAM address:
1. 0x8000_0000 - QEMU and SiFive board
2. 0x40_0000_0000 - Microchip board

So if we mapping frametable to 0 ( not RAM start ) we will loose:
1. 0x8000_0 ( amount of page entries to cover range [0, 0x8000_0000] )
* 64 ( size of struct page_info ) = 32 MB
2. 0x40_0000_0 ( amount of page entries to cover range [0, 0x40_0000_0000] ) * 64 ( size of struct page_info ) = 4 Gb

In terms of available virtual address space for RV-64 we can consider both options as acceptable.

[OPTION 1] If we accepting of loosing 4 Gb of VA then we could implement mfn_to_page() and page_to_mfn() in the following way:

```
   diff --git a/xen/arch/riscv/include/asm/mm.h
   b/xen/arch/riscv/include/asm/mm.h
   index cc4a07a71c..fdac7e0646 100644
   --- a/xen/arch/riscv/include/asm/mm.h
   +++ b/xen/arch/riscv/include/asm/mm.h
   @@ -107,14 +107,11 @@ struct page_info
   
    #define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
   
   -/* PDX of the first page in the frame table. */
   -extern unsigned long frametable_base_pdx;
   -
    /* Convert between machine frame numbers and page-info structures.
*/
    #define mfn_to_page(mfn)                                          
\
   -    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
   +    (frame_table + mfn))
    #define page_to_mfn(pg)                                           
\
   -    pdx_to_mfn((unsigned long)((pg) - frame_table) +
   frametable_base_pdx)
   +    ((unsigned long)((pg) - frame_table))
   
    static inline void *page_to_virt(const struct page_info *pg)
    {
   diff --git a/xen/arch/riscv/mm.c b/xen/arch/riscv/mm.c
   index 9c0fd80588..8f6dbdc699 100644
   --- a/xen/arch/riscv/mm.c
   +++ b/xen/arch/riscv/mm.c
   @@ -15,7 +15,7 @@
    #include <asm/page.h>
    #include <asm/processor.h>
   
   -unsigned long __ro_after_init frametable_base_pdx;
    unsigned long __ro_after_init frametable_virt_end;
   
    struct mmu_desc {
```

[OPTION 2] If we are not accepting of loosing 4 Gb of VA I think that there is no sense to rework that in the following way:

```
   diff --git a/xen/arch/riscv/include/asm/mm.h
   b/xen/arch/riscv/include/asm/mm.h
   index cc4a07a71c..fdac7e0646 100644
   --- a/xen/arch/riscv/include/asm/mm.h
   +++ b/xen/arch/riscv/include/asm/mm.h
   @@ -107,14 +107,11 @@ struct page_info
   
    #define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
   
   -/* PDX of the first page in the frame table. */
   -extern unsigned long frametable_base_pdx;
   -
    /* Convert between machine frame numbers and page-info structures.
*/
    #define mfn_to_page(mfn)                                          
\
   -    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
   +    (frame_table + mfn - FRAMETABLE_BASE_OFFSET))
    #define page_to_mfn(pg)                                           
\
   -    pdx_to_mfn((unsigned long)((pg) - frame_table) +
   frametable_base_pdx)
   +    ((unsigned long)((pg) - frame_table) + FRAMETABLE_BASE_OFFSET)
   
    static inline void *page_to_virt(const struct page_info *pg)
    {
   diff --git a/xen/arch/riscv/mm.c b/xen/arch/riscv/mm.c
   index 9c0fd80588..8f6dbdc699 100644
   --- a/xen/arch/riscv/mm.c
   +++ b/xen/arch/riscv/mm.c
   @@ -15,7 +15,7 @@
    #include <asm/page.h>
    #include <asm/processor.h>
   
   -unsigned long __ro_after_init frametable_base_pdx;
    unsigned long __ro_after_init frametable_virt_end;
   
    struct mmu_desc {
```

And I am not sure that there is any sense in option 2 as basically it the same to have the following macros definition with PDX disabled:

```
   /* Convert between machine frame numbers and page-info structures.
   */
   #define mfn_to_page(mfn)                                           
   \
       (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
   #define page_to_mfn(pg)                                            
   \
       pdx_to_mfn((unsigned long)((pg) - frame_table) +
   frametable_base_pdx)
```

The only sense of option 2 is the case when FRAMETABLE_BASE_OFFSET is equal to 0, so the compiler will generate the code without additional sub/add of FRAMETABLE_BASE_OFFSET.

Could you please clarify if my understanding is correct?

Should we still change the definition of mfn_to_page() and page_to_mfn() with the fact that PDX is disabled? If yes, OPTION_1 is okay or I am missing something?

Thanks in advance.

~ Oleksii

#14 lore raw

Re: [PATCH v12 5/8] xen/riscv: add minimal stuff to mm.h to build full Xen

Quoted 245 lines · Oleksii K.
Quoted 220 lines · Andrew Cooper
Quoted 151 lines · Oleksii K.
Quoted 35 lines · Andrew Cooper
Quoted 2 lines · Oleksii Kurochko

This patch looks like it can go in independently?  Or does it depend
on
having bitops.h working in practice?

However, one very strong suggestion...

Quoted 12 lines
xen/arch/riscv/include/asm/mm.h +4 -0
index 07c7a0abba..cc4a07a71c 100644
@@ -3,11 +3,246 @@
<snip>
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info

structures.

Quoted 8 lines

*/
+#define
mfn_to_page(mfn)                                            \
+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx)) +#define
page_to_mfn(pg)                                             \
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)

Do yourself a favour and not introduce frametable_base_pdx to begin
with.

Quoted 3 lines

Every RISC-V board I can find has things starting from 0 in physical
address space, with RAM starting immediately after.

I checked Linux kernel and grep there:
   [ok@fedora linux-aia]$ grep -Rni "memory@" arch/riscv/boot/dts/ --
   exclude "*.tmp" -I arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive- 2.dtsi:33:
   memory@40000000 { arch/riscv/boot/dts/starfive/jh7100-common.dtsi:28:
memory@80000000
   {
   arch/riscv/boot/dts/microchip/mpfs-sev-kit.dts:49:
ddrc_cache:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:33:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:37:
ddrc_cache_hi:
   memory@1040000000 { arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:34:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:40:
ddrc_cache_hi:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:22:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:27:
ddrc_cache_hi:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:57:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:63:
ddrc_cache_hi:
   memory@1040000000 { arch/riscv/boot/dts/thead/th1520-beaglev-ahead.dts:32:  memory@0 { arch/riscv/boot/dts/thead/th1520-lichee-module-4a.dtsi:14:
   memory@0 { arch/riscv/boot/dts/sophgo/cv1800b-milkv-duo.dts:26:
memory@80000000
   {
   arch/riscv/boot/dts/sophgo/cv1812h.dtsi:12:     memory@80000000 { arch/riscv/boot/dts/sifive/hifive-unmatched-a00.dts:26:
memory@80000000
   { arch/riscv/boot/dts/sifive/hifive-unleashed-a00.dts:25:
memory@80000000
   {
   arch/riscv/boot/dts/canaan/k210.dtsi:82:        sram:
memory@80000000 {

And based on  that majority of supported by Linux kernel boards has RAM
starting not from 0 in physical address space. Am I confusing something?

Quoted 2 lines

Taking the microchip board as an example, RAM actually starts at 0x8000000,

Today we had conversation with the guy from SiFive in xen-devel channel
and he mentioned that they are using "starfive visionfive2 and sifive
unleashed platforms." which based on the grep above has RAM not at 0
address.

Also, QEMU uses 0x8000000.

Quoted 24 lines

 which means that having frametable_base_pdx and assuming it does get set to 0x8000 (which isn't even a certainty, given that I
think
you'll need struct pages covering the PLICs), then what you are trading
off is:

* Saving 32k of virtual address space only (no need to even allocate
memory for this range of the framtable), by
* Having an extra memory load and add/sub in every page <-> mfn conversion, which is a screaming hotpath all over Xen.

It's a terribly short-sighted tradeoff.

32k of VA space might be worth saving in a 32bit build (I personally
wouldn't - especially as there's no need to share Xen's VA space with
guests, given no PV guests on ARM/RISC-V), but it's absolutely not at
all in an a 64bit build with TB of VA space available.

Quoted 6 lines

Even if we do find a board with the first interesting thing in the
frametable starting sufficiently away from 0 that it might be worth
considering this slide, then it should still be Kconfig-able in a similar way to PDX_COMPRESSION.

I found your tradeoffs reasonable, but I don't understand how it will
work if RAM does not start from 0, as the frametable address and RAM
address are linked.
I tried to look at the PDX_COMPRESSION config and couldn't find any "slide" there. Could you please clarify this for me?
If to use this "slide" would it help to avoid the mentioned above tradeoffs?

Quoted 4 lines

One more question: if we decide to go without frametable_base_pdx, would it be sufficient to simply remove mentions of it from the code (
at least, for now )?

There is a relationship between system/host physical addresses (what Xen
called maddr/mfn), and the frametable.  The frametable has one entry per
mfn.

In the most simple case, there's a 1:1 relationship.  i.e. frametable[0]
= maddr(0), frametable[1] = maddr(4k) etc.  This is very simple, and very easy to calculate (page_to_mfn()/mfn_to_page()).

The frametable is one big array.  It starts at a compile-time fixed address, and needs to be long enough to cover everything interesting in
memory.  Therefore it potentially takes a large amount of virtual address space.

However, only interesting maddrs need to have data in the frametable, so
it's fine for the backing RAM to be sparsely allocated/mapped in the frametable virtual addresses.

For 64bit, that's really all you need, because there's always far more
virtual address space than physical RAM in the system, even when you're
looking at TB-scale giant servers.

For 32bit, virtual address space is a limited resource.  (Also to an extent, 64bit x86 with PV guests because we give 98% of the virtual address space to the guest kernel to use.)

There are two tricks to reduce the virtual address space used, but they
both cost performance in fastpaths.

1) PDX Compression.

PDX compression makes a non-linear mfn <-> maddr mapping.  This is for a
usecase where you've got multiple RAM banks which are separated by a large distance (and evenly spaced), then you can "compress" a single range of 0's out of the middle of the system/host physical address.

The cost is that all page <-> mfn conversions need to read two masks and
a shift-count from variables in memory, to split/shift/recombine the address bits.

2) A slide, which is frametable_base_pdx in this context.

When there's a big gap between 0 and the start of something interesting,
you could chop out that range by just subtracting base_pdx.  What qualifies as "big" is subjective, but Qemu starting at 128M certainly does not qualify as big enough to warrant frametable_base_pdx.

This is less expensive that PDX compression.  It only adds one memory read to the fastpath, but it also doesn't save as much virtual address
space as PDX compression.

Quoted 7 lines

When virtual address space is a major constraint (32 bit builds), both
of these techniques are worth doing.  But when there's no constraint on
virtual address space (64 bit builds), there's no reason to use either;
and the performance will definitely improve as a result.

Thanks for such good explanation.

For RISC-V we have PDX config disabled as I haven't seen multiple RAM banks at boards which has hypervisor extension. Thereby mfn_to_pdx() and pdx_to_mfn() do nothing. The same for frametable_base_pdx, in case of PDX disabled, it just an offset ( or a slide ).

IIUUC, you meant that it make sense to map frametable not to the address of where RAM is started, but to 0, so then we don't need this +-frametable_base_pdx. The price for that is loosing of VA space for the range from 0 to RAM start address.

Right now, we are trying to support 3 boards with the following RAM address:
1. 0x8000_0000 - QEMU and SiFive board
2. 0x40_0000_0000 - Microchip board

Quoted 8 lines

So if we mapping frametable to 0 ( not RAM start ) we will loose:
1. 0x8000_0 ( amount of page entries to cover range [0, 0x8000_0000] )
* 64 ( size of struct page_info ) = 32 MB
2. 0x40_0000_0 ( amount of page entries to cover range [0, 0x40_0000_0000] ) * 64 ( size of struct page_info ) = 4 Gb

In terms of available virtual address space for RV-64 we can consider both options as acceptable.

For Qemu and SiFive, 32M is definitely not worth worrying about.

I personally wouldn't worry about Microchip either.  That's 4G of 1T VA space (assuming Sv39).

Quoted 34 lines

[OPTION 1] If we accepting of loosing 4 Gb of VA then we could implement mfn_to_page() and page_to_mfn() in the following way:

```
   diff --git a/xen/arch/riscv/include/asm/mm.h
   b/xen/arch/riscv/include/asm/mm.h
   index cc4a07a71c..fdac7e0646 100644
   --- a/xen/arch/riscv/include/asm/mm.h
   +++ b/xen/arch/riscv/include/asm/mm.h
   @@ -107,14 +107,11 @@ struct page_info
   
    #define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
   
   -/* PDX of the first page in the frame table. */
   -extern unsigned long frametable_base_pdx;
   -
    /* Convert between machine frame numbers and page-info structures.
*/
    #define mfn_to_page(mfn)                                          
\
   -    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
   +    (frame_table + mfn))
    #define page_to_mfn(pg)                                           
\
   -    pdx_to_mfn((unsigned long)((pg) - frame_table) +
   frametable_base_pdx)
   +    ((unsigned long)((pg) - frame_table))
   
    static inline void *page_to_virt(const struct page_info *pg)
    {
   diff --git a/xen/arch/riscv/mm.c b/xen/arch/riscv/mm.c
   index 9c0fd80588..8f6dbdc699 100644
   --- a/xen/arch/riscv/mm.c
   +++ b/xen/arch/riscv/mm.c
   @@ -15,7 +15,7 @@
Quoted 8 lines
    #include <asm/page.h>
    #include <asm/processor.h>
   
   -unsigned long __ro_after_init frametable_base_pdx;
    unsigned long __ro_after_init frametable_virt_end;
   
    struct mmu_desc {
```

I firmly recommend option 1, especially at this point.

If specific boards really have a problem with losing 4G of VA space, then option 2 can be added easily at a later point.

That said, I'd think carefully about doing option 2.  Even subtracting a constant - which is far better than subtracting a variable - is still extra overhead in fastpaths.  Option 2 needs careful consideration on a board-by-board case.

~Andrew

author
#15 lore raw

Re: [PATCH v12 5/8] xen/riscv: add minimal stuff to mm.h to build full Xen

Quoted 356 lines · Andrew Cooper
Quoted 299 lines · Oleksii K.
Quoted 261 lines · Andrew Cooper
Quoted 173 lines · Oleksii K.
Quoted 39 lines · Andrew Cooper
Quoted 3 lines · Oleksii Kurochko

Signed-off-by: Oleksii Kurochko
<[email protected]>

This patch looks like it can go in independently?  Or does it depend
on
having bitops.h working in practice?

However, one very strong suggestion...

Quoted 15 lines
xen/arch/riscv/include/asm/mm.h +4 -0
index 07c7a0abba..cc4a07a71c 100644
@@ -3,11 +3,246 @@
<snip>
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info

structures.
*/
+#define
mfn_to_page(mfn)

Quoted 8 lines

\
+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
+#define
page_to_mfn(pg)
\
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)

Do yourself a favour and not introduce frametable_base_pdx to begin
with.

Quoted 3 lines

Every RISC-V board I can find has things starting from 0 in physical
address space, with RAM starting immediately after.

I checked Linux kernel and grep there:
   [ok@fedora linux-aia]$ grep -Rni "memory@" arch/riscv/boot/dts/
--
   exclude "*.tmp" -I arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive- 2.dtsi:33:
   memory@40000000 { arch/riscv/boot/dts/starfive/jh7100-common.dtsi:28:
memory@80000000
   { arch/riscv/boot/dts/microchip/mpfs-sev-kit.dts:49:
ddrc_cache:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:33:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-m100pfsevp.dts:37:
ddrc_cache_hi:
   memory@1040000000 { arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:34:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-tysom-m.dts:40:
ddrc_cache_hi:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:22:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-polarberry.dts:27:
ddrc_cache_hi:
   memory@1000000000 { arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:57:
ddrc_cache_lo:
   memory@80000000 { arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts:63:
ddrc_cache_hi:
   memory@1040000000 { arch/riscv/boot/dts/thead/th1520-beaglev-ahead.dts:32:
memory@0
{
   arch/riscv/boot/dts/thead/th1520-lichee-module- 4a.dtsi:14:
   memory@0 { arch/riscv/boot/dts/sophgo/cv1800b-milkv-duo.dts:26:
memory@80000000
   {
   arch/riscv/boot/dts/sophgo/cv1812h.dtsi:12:
memory@80000000
{ arch/riscv/boot/dts/sifive/hifive-unmatched-a00.dts:26:
memory@80000000
   { arch/riscv/boot/dts/sifive/hifive-unleashed-a00.dts:25:
memory@80000000
   {
   arch/riscv/boot/dts/canaan/k210.dtsi:82:        sram:
memory@80000000 {

And based on  that majority of supported by Linux kernel boards has
RAM
starting not from 0 in physical address space. Am I confusing something?

Quoted 3 lines

Taking the microchip board as an example, RAM actually starts at
0x8000000,

Today we had conversation with the guy from SiFive in xen-devel channel
and he mentioned that they are using "starfive visionfive2 and sifive
unleashed platforms." which based on the grep above has RAM not at
0
address.

Also, QEMU uses 0x8000000.

Quoted 29 lines

 which means that having frametable_base_pdx and assuming it does get set to 0x8000 (which isn't even a certainty, given that
I
think
you'll need struct pages covering the PLICs), then what you are
trading
off is:

* Saving 32k of virtual address space only (no need to even allocate
memory for this range of the framtable), by
* Having an extra memory load and add/sub in every page <-> mfn
conversion, which is a screaming hotpath all over Xen.

It's a terribly short-sighted tradeoff.

32k of VA space might be worth saving in a 32bit build (I personally
wouldn't - especially as there's no need to share Xen's VA space
with
guests, given no PV guests on ARM/RISC-V), but it's absolutely
not at
all in an a 64bit build with TB of VA space available.

Quoted 8 lines

Even if we do find a board with the first interesting thing in
the
frametable starting sufficiently away from 0 that it might be worth
considering this slide, then it should still be Kconfig-able in a
similar way to PDX_COMPRESSION.

I found your tradeoffs reasonable, but I don't understand how it
will
work if RAM does not start from 0, as the frametable address and
RAM
address are linked.
I tried to look at the PDX_COMPRESSION config and couldn't find any
"slide" there. Could you please clarify this for me? If to use this "slide" would it help to avoid the mentioned above
tradeoffs?

Quoted 5 lines

One more question: if we decide to go without frametable_base_pdx,
would it be sufficient to simply remove mentions of it from the code (
at least, for now )?

There is a relationship between system/host physical addresses (what
Xen
called maddr/mfn), and the frametable.  The frametable has one entry
per
mfn.

In the most simple case, there's a 1:1 relationship.  i.e. frametable[0]
= maddr(0), frametable[1] = maddr(4k) etc.  This is very simple, and
very easy to calculate (page_to_mfn()/mfn_to_page()).

The frametable is one big array.  It starts at a compile-time fixed
address, and needs to be long enough to cover everything interesting
in
memory.  Therefore it potentially takes a large amount of virtual address space.

However, only interesting maddrs need to have data in the frametable,
so
it's fine for the backing RAM to be sparsely allocated/mapped in the
frametable virtual addresses.

For 64bit, that's really all you need, because there's always far more
virtual address space than physical RAM in the system, even when you're
looking at TB-scale giant servers.

For 32bit, virtual address space is a limited resource.  (Also to an
extent, 64bit x86 with PV guests because we give 98% of the virtual
address space to the guest kernel to use.)

There are two tricks to reduce the virtual address space used, but
they
both cost performance in fastpaths.

1) PDX Compression.

PDX compression makes a non-linear mfn <-> maddr mapping.  This is
for a
usecase where you've got multiple RAM banks which are separated by a
large distance (and evenly spaced), then you can "compress" a single
range of 0's out of the middle of the system/host physical address.

The cost is that all page <-> mfn conversions need to read two masks
and
a shift-count from variables in memory, to split/shift/recombine the
address bits.

2) A slide, which is frametable_base_pdx in this context.

When there's a big gap between 0 and the start of something interesting,
you could chop out that range by just subtracting base_pdx.  What qualifies as "big" is subjective, but Qemu starting at 128M certainly
does not qualify as big enough to warrant frametable_base_pdx.

This is less expensive that PDX compression.  It only adds one memory
read to the fastpath, but it also doesn't save as much virtual address
space as PDX compression.

Quoted 8 lines

When virtual address space is a major constraint (32 bit builds), both
of these techniques are worth doing.  But when there's no constraint
on
virtual address space (64 bit builds), there's no reason to use either;
and the performance will definitely improve as a result.

Thanks for such good explanation.

For RISC-V we have PDX config disabled as I haven't seen multiple RAM
banks at boards which has hypervisor extension. Thereby mfn_to_pdx()
and pdx_to_mfn() do nothing. The same for frametable_base_pdx, in case
of PDX disabled, it just an offset ( or a slide ).

IIUUC, you meant that it make sense to map frametable not to the address of where RAM is started, but to 0, so then we don't need this
+-frametable_base_pdx. The price for that is loosing of VA space for
the range from 0 to RAM start address.

Right now, we are trying to support 3 boards with the following RAM address:
1. 0x8000_0000 - QEMU and SiFive board
2. 0x40_0000_0000 - Microchip board

So if we mapping frametable to 0 ( not RAM start ) we will loose:
1. 0x8000_0 ( amount of page entries to cover range [0, 0x8000_0000] )
* 64 ( size of struct page_info ) = 32 MB
2. 0x40_0000_0 ( amount of page entries to cover range [0, 0x40_0000_0000] ) * 64 ( size of struct page_info ) = 4 Gb

Quoted 3 lines

In terms of available virtual address space for RV-64 we can consider
both options as acceptable.

For Qemu and SiFive, 32M is definitely not worth worrying about.

I personally wouldn't worry about Microchip either.  That's 4G of 1T VA
space (assuming Sv39).

Quoted 37 lines

[OPTION 1] If we accepting of loosing 4 Gb of VA then we could implement mfn_to_page() and page_to_mfn() in the following way:

```
   diff --git a/xen/arch/riscv/include/asm/mm.h
   b/xen/arch/riscv/include/asm/mm.h
   index cc4a07a71c..fdac7e0646 100644
   --- a/xen/arch/riscv/include/asm/mm.h
   +++ b/xen/arch/riscv/include/asm/mm.h
   @@ -107,14 +107,11 @@ struct page_info
   
    #define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
   
   -/* PDX of the first page in the frame table. */
   -extern unsigned long frametable_base_pdx;
   -
    /* Convert between machine frame numbers and page-info
structures.
*/
    #define
mfn_to_page(mfn)                                          
\
   -    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
   +    (frame_table + mfn))
    #define
page_to_mfn(pg)                                           
\
   -    pdx_to_mfn((unsigned long)((pg) - frame_table) +
   frametable_base_pdx)
   +    ((unsigned long)((pg) - frame_table))
   
    static inline void *page_to_virt(const struct page_info *pg)
    {
   diff --git a/xen/arch/riscv/mm.c b/xen/arch/riscv/mm.c
   index 9c0fd80588..8f6dbdc699 100644
   --- a/xen/arch/riscv/mm.c
   +++ b/xen/arch/riscv/mm.c
   @@ -15,7 +15,7 @@
Quoted 8 lines
    #include <asm/page.h>
    #include <asm/processor.h>
   
   -unsigned long __ro_after_init frametable_base_pdx;
    unsigned long __ro_after_init frametable_virt_end;
   
    struct mmu_desc {
```
Quoted 1 line

I firmly recommend option 1, especially at this point.

Jan, as you gave your Acked before, don't you mind to define mfn_to_page() and page_to_mfn as mentioned above ( Option 1 )?

~ Oleksi

Quoted 12 lines

If specific boards really have a problem with losing 4G of VA space, then option 2 can be added easily at a later point.

That said, I'd think carefully about doing option 2.  Even subtracting a
constant - which is far better than subtracting a variable - is still extra overhead in fastpaths.  Option 2 needs careful consideration on a
board-by-board case.

~Andrew

#16 lore raw

Re: [PATCH v12 5/8] xen/riscv: add minimal stuff to mm.h to build full Xen

Quoted 48 lines · Oleksii K.
Quoted 47 lines · Andrew Cooper
Quoted 45 lines · Oleksii K.

[OPTION 1] If we accepting of loosing 4 Gb of VA then we could implement mfn_to_page() and page_to_mfn() in the following way:

```
   diff --git a/xen/arch/riscv/include/asm/mm.h
   b/xen/arch/riscv/include/asm/mm.h
   index cc4a07a71c..fdac7e0646 100644
   --- a/xen/arch/riscv/include/asm/mm.h
   +++ b/xen/arch/riscv/include/asm/mm.h
   @@ -107,14 +107,11 @@ struct page_info
   
    #define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
   
   -/* PDX of the first page in the frame table. */
   -extern unsigned long frametable_base_pdx;
   -
    /* Convert between machine frame numbers and page-info
structures.
*/
    #define
mfn_to_page(mfn)                                          
\
   -    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
   +    (frame_table + mfn))
    #define
page_to_mfn(pg)                                           
\
   -    pdx_to_mfn((unsigned long)((pg) - frame_table) +
   frametable_base_pdx)
   +    ((unsigned long)((pg) - frame_table))
   
    static inline void *page_to_virt(const struct page_info *pg)
    {
   diff --git a/xen/arch/riscv/mm.c b/xen/arch/riscv/mm.c
   index 9c0fd80588..8f6dbdc699 100644
   --- a/xen/arch/riscv/mm.c
   +++ b/xen/arch/riscv/mm.c
   @@ -15,7 +15,7 @@
    #include <asm/page.h>
    #include <asm/processor.h>
   
   -unsigned long __ro_after_init frametable_base_pdx;
    unsigned long __ro_after_init frametable_virt_end;
   
    struct mmu_desc {
```
Quoted 3 lines
Quoted 1 line

I firmly recommend option 1, especially at this point.

Jan, as you gave your Acked before, don't you mind to define mfn_to_page() and page_to_mfn as mentioned above ( Option 1 )?

No, I don't mind. And please feel free to keep my ack if no other significant changes are made.

Jan

author
#18 lore raw

Re: [PATCH v12 5/8] xen/riscv: add minimal stuff to mm.h to build full Xen

In reply to Andrew Cooper
Quoted 25 lines · Andrew Cooper
Quoted 2 lines · Oleksii Kurochko

This patch looks like it can go in independently?  Or does it depend on
having bitops.h working in practice?

However, one very strong suggestion...

Quoted 14 lines
xen/arch/riscv/include/asm/mm.h +4 -0
index 07c7a0abba..cc4a07a71c 100644
@@ -3,11 +3,246 @@
<snip>
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info structures.

*/
+#define
mfn_to_page(mfn)                                            \

Quoted 8 lines
Quoted 5 lines

+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx)) +#define
page_to_mfn(pg)                                             \
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)

Do yourself a favour and not introduce frametable_base_pdx to begin with.

To drop frametable_base_pdx if the following changes will be enough:
diff --git a/xen/arch/riscv/include/asm/mm.h b/xen/arch/riscv/include/asm/mm.h
index cc4a07a71c..fdac7e0646 100644
--- a/xen/arch/riscv/include/asm/mm.h
+++ b/xen/arch/riscv/include/asm/mm.h
@@ -107,14 +107,11 @@ struct page_info

    #define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)

-/* PDX of the first page in the frame table. */ -extern unsigned long frametable_base_pdx; -

    /* Convert between machine frame numbers and page-info structures. */
    #define mfn_to_page(mfn)                                            \
   -    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
   +    (frame_table + mfn - FRAMETABLE_BASE_OFFSET))
    #define page_to_mfn(pg)                                             \

- pdx_to_mfn((unsigned long)((pg) - frame_table) +
frametable_base_pdx)
+ ((unsigned long)((pg) - frame_table) + FRAMETABLE_BASE_OFFSET)

static inline void *page_to_virt(const struct page_info *pg) {
diff --git a/xen/arch/riscv/mm.c b/xen/arch/riscv/mm.c index 9c0fd80588..8f6dbdc699 100644
--- a/xen/arch/riscv/mm.c
+++ b/xen/arch/riscv/mm.c
@@ -15,7 +15,7 @@

    #include <asm/page.h>
    #include <asm/processor.h>
   -unsigned long __ro_after_init frametable_base_pdx;
    unsigned long __ro_after_init frametable_virt_end;
    
    struct mmu_desc {
Quoted 13 lines

Every RISC-V board I can find has things starting from 0 in physical address space, with RAM starting immediately after.

Taking the microchip board as an example, RAM actually starts at 0x8000000, which means that having frametable_base_pdx and assuming it
does get set to 0x8000 (which isn't even a certainty, given that I think
you'll need struct pages covering the PLICs), then what you are trading
off is:

Quoted 4 lines

* Saving 32k of virtual address space only (no need to even allocate memory for this range of the framtable), by
* Having an extra memory load and add/sub in every page <-> mfn conversion, which is a screaming hotpath all over Xen.

Are you referring here to mfn_to_pdx() used in mfn_to_page() and pdx_to_mfn() in page_to_mfn()?

My expectation was that when CONFIG_PDX_COMPRESSION is disabled then this macros doesn't do anything:
/* pdx<->pfn == identity */
#define pdx_to_pfn(x) (x)
#define pfn_to_pdx(x) (x)

Quoted 7 lines

It's a terribly short-sighted tradeoff.

32k of VA space might be worth saving in a 32bit build (I personally wouldn't - especially as there's no need to share Xen's VA space with guests, given no PV guests on ARM/RISC-V), but it's absolutely not at all in an a 64bit build with TB of VA space available.

Why 32k? If RAM is started at 0x8000_0000 then we have to cover 0x80000 entries, and the size of it is 0x8_0000 = 524288 * 64 = 33554432, so it is 32 Mb.
Am I confusing something?

P.S.: Should I map this start 32 mb? Or it is enough to have a slide ( FRAMETABLE_BASE_OFFSET ).

~ Oleksii

Quoted 11 lines

Even if we do find a board with the first interesting thing in the frametable starting sufficiently away from 0 that it might be worth considering this slide, then it should still be Kconfig-able in a similar way to PDX_COMPRESSION.

You don't want to be penalising everyone because of a theoretical/weird
corner case.

~Andrew

#19 lore raw

PATCH v12 6/8 xen/riscv: add minimal amount of stubs to build full Xen

In reply to Oleksii Kurochko

---
Changes in V7-V12:
- Only rebase was done.
---
Changes in V6:
- update the commit in stubs.c around /* ... common/irq.c ... */
- add Acked-by: Jan Beulich <[email protected]>
---
Changes in V5:
- drop unrelated changes
- assert_failed("unimplmented...") change to BUG_ON()
---
Changes in V4:
- added new stubs which are necessary for compilation after rebase: __cpu_up(), __cpu_disable(), __cpu_die() from smpboot.c
- back changes related to printk() in early_printk() as they should be removed in the next patch to avoid compilation error.
- update definition of cpu_khz: __read_mostly -> __ro_after_init.
- drop vm_event_reset_vmtrace(). It is defibed in asm-generic/vm_event.h.
- move vm_event_*() functions from stubs.c to riscv/vm_event.c.
- s/BUG/BUG_ON("unimplemented") in stubs.c
- back irq_actor_none() and irq_actor_none() as common/irq.c isn't compiled at this moment, so this function are needed to avoid compilation error.
- defined max_page to avoid compilation error, it will be removed as soon as common/page_alloc.c will be compiled.
---
Changes in V3:
- code style fixes.
- update attribute for frametable_base_pdx and frametable_virt_end to __ro_after_init. insteaf of read_mostly.
- use BUG() instead of assert_failed/WARN for newly introduced stubs.
- drop "#include <public/vm_event.h>" in stubs.c and use forward declaration instead.
- drop ack_node() and end_node() as they aren't used now.
---
Changes in V2:
- define udelay stub
- remove 'select HAS_PDX' from RISC-V Kconfig because of
https://lore.kernel.org/xen-devel/[email protected]/

---
 xen/arch/riscv/Makefile |   1 +
 xen/arch/riscv/mm.c     |  50 +++++
 xen/arch/riscv/setup.c  |   8 +
 xen/arch/riscv/stubs.c  | 439 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/riscv/traps.c  |  25 +++
 5 files changed, 523 insertions(+)
 create mode 100644 xen/arch/riscv/stubs.c
xen/arch/riscv/Makefile +1 -0
index 1ed1a8369b..60afbc0ad9 100644
@@ -4,6 +4,7 @@ obj-y += mm.o
 obj-$(CONFIG_RISCV_64) += riscv64/
 obj-y += sbi.o
 obj-y += setup.o
+obj-y += stubs.o
 obj-y += traps.o
 obj-y += vm_event.o
 
xen/arch/riscv/mm.c +50 -0

64 lines not on this page · raw message

xen/arch/riscv/setup.c +8 -0
index 98a94c4c48..8bb5bdb2ae 100644
@@ -1,11 +1,19 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 
+#include <xen/bug.h>
 #include <xen/compile.h>
 #include <xen/init.h>
 #include <xen/mm.h>
 
+#include <public/version.h>
+
 #include <asm/early_printk.h>
 
+void arch_get_xen_caps(xen_capabilities_info_t *info)
+{
+    BUG_ON("unimplemented");
+}
+
 /* Xen stack for bringing up the first CPU. */
 unsigned char __initdata cpu0_boot_stack[STACK_SIZE]
     __aligned(STACK_SIZE);
xen/arch/riscv/stubs.c new file +439 -0

439 lines not on this page · raw message

xen/arch/riscv/traps.c +25 -0
index ccd3593f5a..5415cf8d90 100644
@@ -4,6 +4,10 @@
  *
  * RISC-V Trap handlers
  */
+
+#include <xen/lib.h>
+#include <xen/sched.h>
+
 #include <asm/processor.h>
 #include <asm/traps.h>
 
@@ -11,3 +15,24 @@ void do_trap(struct cpu_user_regs *cpu_regs)
 {
     die();
 }
+
+void vcpu_show_execution_state(struct vcpu *v)
+{
+    BUG_ON("unimplemented");
+}
+
+void show_execution_state(const struct cpu_user_regs *regs)
+{
+    printk("implement show_execution_state(regs)\n");
+}
+
+void arch_hypercall_tasklet_result(struct vcpu *v, long res)
+{
+    BUG_ON("unimplemented");
+}
+
+enum mc_disposition arch_do_multicall_call(struct mc_state *state)
+{
+    BUG_ON("unimplemented");
+    return mc_continue;
+}
Signature · 1 line
2.45.0
#20 lore raw

Re: [PATCH v12 6/8] xen/riscv: add minimal amount of stubs to build full Xen

xen/arch/riscv/stubs.c line 399

Quoted 13 lines · Oleksii Kurochko
xen/arch/riscv/stubs.c new file +6 -0
new file mode 100644
index 0000000000..8285bcffef
@@ -0,0 +1,439 @@
<snip>
+void udelay(unsigned long usecs)
+{
+    BUG_ON("unimplemented");
+}
+
+/* guest_access.h */ 

You've got trailing whitespace here.

Also, this patch is (still) a shocking indictment on how bad our common vs arch split really is.

~Andrew

#21 lore raw

Re: [PATCH v12 6/8] xen/riscv: add minimal amount of stubs to build full Xen

In reply to Oleksii Kurochko
Quoted 12 lines · Oleksii Kurochko
xen/arch/riscv/mm.c +1 -0
index fe3a43be20..2c3fb7d72e 100644
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 
+#include <xen/bug.h>
 #include <xen/cache.h>
 #include <xen/compiler.h>
 #include <xen/init.h>

xen/arch/riscv/mm.c line 19

Quoted 5 lines
xen/arch/riscv/mm.c +2 -0
@@ -14,6 +15,9 @@
 #include <asm/page.h>
 #include <asm/processor.h>
 
+unsigned long __ro_after_init frametable_base_pdx;
+unsigned long __ro_after_init frametable_virt_end;

Along with removing base_pdx, virt_end appears to be unique to ARM before and after the series.  I'd recommend dropping it too.

~Andrew

#22 lore raw

PATCH v12 7/8 xen/riscv: enable full Xen build

In reply to Oleksii Kurochko

---
At least this patch cann't be merged w/o Andrew's patch series is merged as ffs related functions are used from that patch series:
https://lore.kernel.org/xen-devel/[email protected]/T/#t
---
Changes in V5-V12:
- Nothing changed. Only rebase.
- Add the footer after Signed-off section.
---
Changes in V4:
- drop stubs for irq_actor_none() and irq_actor_none() as common/irq.c is compiled now.
- drop defintion of max_page in stubs.c as common/page_alloc.c is compiled now.
- drop printk() related changes in riscv/early_printk.c as common version will be used.
---
Changes in V3:
- Reviewed-by: Jan Beulich <[email protected]>
- unrealted change dropped in tiny64_defconfig
---
Changes in V2:
- Nothing changed. Only rebase.

---
 xen/arch/riscv/Makefile       |  16 +++-
 xen/arch/riscv/arch.mk        |   4 -
 xen/arch/riscv/early_printk.c | 168 ----------------------------------
 xen/arch/riscv/stubs.c        |  24 -----
 4 files changed, 15 insertions(+), 197 deletions(-)
xen/arch/riscv/Makefile +15 -1
index 60afbc0ad9..81b77b13d6 100644
@@ -12,10 +12,24 @@ $(TARGET): $(TARGET)-syms
 	$(OBJCOPY) -O binary -S $< $@
 
 $(TARGET)-syms: $(objtree)/prelink.o $(obj)/xen.lds
-	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) -o $@
+	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< \
+	    $(objtree)/common/symbols-dummy.o -o $(dot-target).0
+	$(NM) -pa --format=sysv $(dot-target).0 \
+		| $(objtree)/tools/symbols $(all_symbols) --sysv --sort \
+		> $(dot-target).0.S
+	$(MAKE) $(build)=$(@D) $(dot-target).0.o
+	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< \
+	    $(dot-target).0.o -o $(dot-target).1
+	$(NM) -pa --format=sysv $(dot-target).1 \
+		| $(objtree)/tools/symbols $(all_symbols) --sysv --sort \
+		> $(dot-target).1.S
+	$(MAKE) $(build)=$(@D) $(dot-target).1.o
+	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) \
+	    $(dot-target).1.o -o $@
 	$(NM) -pa --format=sysv $@ \
 		| $(objtree)/tools/symbols --all-symbols --xensyms --sysv --sort \
 		> [email protected]
+	rm -f $(@D)/.$(@F).[0-9]*
 
 $(obj)/xen.lds: $(src)/xen.lds.S FORCE
 	$(call if_changed_dep,cpp_lds_S)
xen/arch/riscv/arch.mk +0 -4
index 8c071aff65..17827c302c 100644
@@ -38,7 +38,3 @@ extensions := $(subst $(space),,$(extensions))
 # -mcmodel=medlow would force Xen into the lower half.
 
 CFLAGS += $(riscv-generic-flags)$(extensions) -mstrict-align -mcmodel=medany
-
-# TODO: Drop override when more of the build is working
-override ALL_OBJS-y = arch/$(SRCARCH)/built_in.o
-override ALL_LIBS-y =
xen/arch/riscv/early_printk.c +0 -168

171 lines not on this page · raw message

xen/arch/riscv/stubs.c +0 -24
index 8285bcffef..bda35fc347 100644
@@ -24,12 +24,6 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
 
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
-/*
- * max_page is defined in page_alloc.c which isn't complied for now.
- * definition of max_page will be remove as soon as page_alloc is built.
- */
-unsigned long __read_mostly max_page;
-
 /* time.c */
 
 unsigned long __ro_after_init cpu_khz;  /* CPU clock frequency in kHz. */
@@ -419,21 +413,3 @@ void __cpu_die(unsigned int cpu)
 {
     BUG_ON("unimplemented");
 }
-
-/*
- * The following functions are defined in common/irq.c, but common/irq.c isn't
- * built for now. These changes will be removed there when common/irq.c is
- * ready.
- */
-
-void cf_check irq_actor_none(struct irq_desc *desc)
-{
-    BUG_ON("unimplemented");
-}
-
-unsigned int cf_check irq_startup_none(struct irq_desc *desc)
-{
-    BUG_ON("unimplemented");
-
-    return 0;
-}
Signature · 1 line
2.45.0
#23 lore raw

Re: [PATCH v12 7/8] xen/riscv: enable full Xen build

Quoted 29 lines · Oleksii Kurochko
xen/arch/riscv/stubs.c +0 -14
index 8285bcffef..bda35fc347 100644
@@ -24,12 +24,6 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
 
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
-/*
- * max_page is defined in page_alloc.c which isn't complied for now.
- * definition of max_page will be remove as soon as page_alloc is built.
- */
-unsigned long __read_mostly max_page;
-
 /* time.c */
 
 unsigned long __ro_after_init cpu_khz;  /* CPU clock frequency in kHz. */
@@ -419,21 +413,3 @@ void __cpu_die(unsigned int cpu)
 {
     BUG_ON("unimplemented");
 }
-
-/*
- * The following functions are defined in common/irq.c, but common/irq.c isn't
- * built for now. These changes will be removed there when common/irq.c is
- * ready.
- */
-
-void cf_check irq_actor_none(struct irq_desc *desc)

xen/arch/riscv/stubs.c void __cpu_die(unsigned int cpu) · line 439

Quoted 10 lines
xen/arch/riscv/stubs.c +0 -10
-{
-    BUG_ON("unimplemented");
-}
-
-unsigned int cf_check irq_startup_none(struct irq_desc *desc)
-{
-    BUG_ON("unimplemented");
-
-    return 0;
-}

All 3 of these are introduced in the previous patch and deleted again here.  Looks like a rebasing accident.

(This patch has to be the final one touching build related things, so I can't simply fix it on commit)

~Andrew

author
#24 lore raw

Re: [PATCH v12 7/8] xen/riscv: enable full Xen build

Quoted 41 lines · Andrew Cooper
Quoted 40 lines · Oleksii Kurochko
xen/arch/riscv/stubs.c +0 -0
index 8285bcffef..bda35fc347 100644
@@ -24,12 +24,6 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t,
 cpu_core_mask);


 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
-/*
- * max_page is defined in page_alloc.c which isn't complied for now.
- * definition of max_page will be remove as soon as page_alloc is built.
- */

xen/arch/riscv/stubs.c +0 -2
-unsigned long __read_mostly max_page;
-

 /* time.c */

 unsigned long __ro_after_init cpu_khz;  /* CPU clock frequency in kHz. */

xen/arch/riscv/stubs.c +0 -3
@@ -419,21 +413,3 @@ void __cpu_die(unsigned int cpu)
  {
      BUG_ON("unimplemented");
  }
-
-/*
- * The following functions are defined in common/irq.c, but

common/irq.c isn't
- * built for now. These changes will be removed there when common/irq.c is
- * ready.
- */

xen/arch/riscv/stubs.c +0 -7
-
-void cf_check irq_actor_none(struct irq_desc *desc)
-{
-    BUG_ON("unimplemented");
-}
-
-unsigned int cf_check irq_startup_none(struct irq_desc *desc)
Quoted 8 lines

xen/arch/riscv/stubs.c void __cpu_die(unsigned int cpu) · line 439

Quoted 5 lines
xen/arch/riscv/stubs.c +0 -5
-{
-    BUG_ON("unimplemented");
-
-    return 0;
-}

All 3 of these are introduced in the previous patch and deleted again here.  Looks like a rebasing accident.

Not really.

This was done to avoid build failures for RISC-V. If the HEAD is on the previous patch where these changes are introduced and then we just drop them, it will lead to a linkage error because these functions are defined in common/irq.c, which isn't built for RISC-V if the HEAD is on the previous patch:
/build/xen/arch/riscv/entry.S:86: undefined reference to `max_page'
riscv64-linux-gnu-ld: prelink.o:(.rodata+0x8): undefined reference to `irq_startup_none'
riscv64-linux-gnu-ld: prelink.o:(.rodata+0x10): undefined reference to `irq_actor_none'
riscv64-linux-gnu-ld: prelink.o:(.rodata+0x18): undefined reference to `irq_actor_none'
riscv64-linux-gnu-ld: prelink.o:(.rodata+0x20): undefined reference to `irq_actor_none'
riscv64-linux-gnu-ld: xen-syms: hidden symbol `irq_actor_none' isn't defined

That is why these stubs were introduced in the previous patch (because common/irq.c isn't built at that moment) and are removed in this patch (since at the moment of this patch, common/irq.c is now being built).

~ Oleksii

Quoted 6 lines

(This patch has to be the final one touching build related things, so I
can't simply fix it on commit)

~Andrew

#25 lore raw

Re: [PATCH v12 7/8] xen/riscv: enable full Xen build

Quoted 49 lines · Oleksii K.
Quoted 40 lines · Andrew Cooper
Quoted 39 lines · Oleksii Kurochko
xen/arch/riscv/stubs.c +0 -0
index 8285bcffef..bda35fc347 100644
@@ -24,12 +24,6 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t,
 cpu_core_mask);


 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
-/*
- * max_page is defined in page_alloc.c which isn't complied for now.
- * definition of max_page will be remove as soon as page_alloc is built.
- */

xen/arch/riscv/stubs.c +0 -2
-unsigned long __read_mostly max_page;
-

 /* time.c */

 unsigned long __ro_after_init cpu_khz;  /* CPU clock frequency in kHz. */

xen/arch/riscv/stubs.c +0 -3
@@ -419,21 +413,3 @@ void __cpu_die(unsigned int cpu)
  {
      BUG_ON("unimplemented");
  }
-
-/*
- * The following functions are defined in common/irq.c, but

common/irq.c isn't
- * built for now. These changes will be removed there when common/irq.c is
- * ready.
- */

xen/arch/riscv/stubs.c +0 -6
-
-void cf_check irq_actor_none(struct irq_desc *desc)
-{
-    BUG_ON("unimplemented");
-}
-
Quoted 8 lines

xen/arch/riscv/stubs.c void __cpu_die(unsigned int cpu) · line 439

Quoted 6 lines
xen/arch/riscv/stubs.c +0 -6
-unsigned int cf_check irq_startup_none(struct irq_desc *desc)
-{
-    BUG_ON("unimplemented");
-
-    return 0;
-}

All 3 of these are introduced in the previous patch and deleted again here.  Looks like a rebasing accident.

Quoted 8 lines

Not really.

This was done to avoid build failures for RISC-V. If the HEAD is on the previous patch where these changes are introduced and then we just drop them, it will lead to a linkage error because these functions are defined in common/irq.c, which isn't built for RISC-V if the HEAD is on the previous patch:
/build/xen/arch/riscv/entry.S:86: undefined reference to `max_page'

Nothing in the series touches entry.S, so I'm not sure what this is complaining about.

The stub for get_upper_mfn_bound() references max_page, but it's only used in a SYSCTL so you can avoid the problem with a BUG_ON().

BTW, you also don't need a return after a BUG_ON(). __builtin_unreachable() takes care of everything properly for you.

Quoted 11 lines
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x8): undefined reference to
   `irq_startup_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x10): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x18): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x20): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: xen-syms: hidden symbol `irq_actor_none' isn't
   defined
Quoted 3 lines

That is why these stubs were introduced in the previous patch (because common/irq.c isn't built at that moment) and are removed in this patch (since at the moment of this patch, common/irq.c is now being built).

These OTOH are a side effect of how no_irq_type works, which is horrifying, and not something I can unsee.

I'll see about fixing it, because I really can't bare to leave it like this...

~Andrew

author
#26 lore raw

Re: [PATCH v12 7/8] xen/riscv: enable full Xen build

Quoted 67 lines · Andrew Cooper
Quoted 55 lines · Oleksii K.
Quoted 44 lines · Andrew Cooper
Quoted 43 lines · Oleksii Kurochko
xen/arch/riscv/stubs.c +0 -0
index 8285bcffef..bda35fc347 100644
@@ -24,12 +24,6 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t,
 cpu_core_mask);


 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
-/*
- * max_page is defined in page_alloc.c which isn't complied for
now.
- * definition of max_page will be remove as soon as page_alloc is
built.
- */

xen/arch/riscv/stubs.c +0 -2
-unsigned long __read_mostly max_page;
-

 /* time.c */

 unsigned long __ro_after_init cpu_khz;  /* CPU clock frequency in
kHz. */

xen/arch/riscv/stubs.c +0 -3
@@ -419,21 +413,3 @@ void __cpu_die(unsigned int cpu)
  {
      BUG_ON("unimplemented");
  }
-
-/*
- * The following functions are defined in common/irq.c, but

common/irq.c isn't
- * built for now. These changes will be removed there when common/irq.c is
- * ready.
- */

xen/arch/riscv/stubs.c +0 -7
-
-void cf_check irq_actor_none(struct irq_desc *desc)
-{
-    BUG_ON("unimplemented");
-}
-
-unsigned int cf_check irq_startup_none(struct irq_desc *desc)
Quoted 8 lines

xen/arch/riscv/stubs.c void __cpu_die(unsigned int cpu) · line 439

Quoted 5 lines
xen/arch/riscv/stubs.c +0 -5
-{
-    BUG_ON("unimplemented");
-
-    return 0;
-}

All 3 of these are introduced in the previous patch and deleted again
here.  Looks like a rebasing accident.

Not really.

Quoted 10 lines

This was done to avoid build failures for RISC-V. If the HEAD is on the
previous patch where these changes are introduced and then we just drop
them, it will lead to a linkage error because these functions are defined in common/irq.c, which isn't built for RISC-V if the HEAD is on
the previous patch:
   /build/xen/arch/riscv/entry.S:86: undefined reference to `max_page'

Quoted 5 lines

Nothing in the series touches entry.S, so I'm not sure what this is complaining about.

The stub for get_upper_mfn_bound() references max_page, but it's only used in a SYSCTL so you can avoid the problem with a BUG_ON().

I didn't get how I can use BUG_ON() with declaration of variable to avoid the compilation issue the undefined reference to max_page?

Quoted 28 lines

BTW, you also don't need a return after a BUG_ON(). __builtin_unreachable() takes care of everything properly for you.

Quoted 16 lines

   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x8): undefined reference to
   `irq_startup_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x10): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x18): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x20): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: xen-syms: hidden symbol `irq_actor_none' isn't
   defined

Quoted 6 lines

That is why these stubs were introduced in the previous patch (because
common/irq.c isn't built at that moment) and are removed in this patch
(since at the moment of this patch, common/irq.c is now being built).

Quoted 6 lines

These OTOH are a side effect of how no_irq_type works, which is horrifying, and not something I can unsee.

I'll see about fixing it, because I really can't bare to leave it like
this...

I am really sorry, but I don't understand should I deal with this somehow now or the provided changes are okay?

~ Oleksii

#27 lore raw

Re: [PATCH v12 7/8] xen/riscv: enable full Xen build

Quoted 67 lines · Oleksii K.
Quoted 66 lines · Andrew Cooper
Quoted 65 lines · Oleksii K.
Quoted 44 lines · Andrew Cooper
Quoted 43 lines · Oleksii Kurochko
xen/arch/riscv/stubs.c +0 -0
index 8285bcffef..bda35fc347 100644
@@ -24,12 +24,6 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t,
 cpu_core_mask);


 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
-/*
- * max_page is defined in page_alloc.c which isn't complied for
now.
- * definition of max_page will be remove as soon as page_alloc is
built.
- */

xen/arch/riscv/stubs.c +0 -2
-unsigned long __read_mostly max_page;
-

 /* time.c */

 unsigned long __ro_after_init cpu_khz;  /* CPU clock frequency in
kHz. */

xen/arch/riscv/stubs.c +0 -3
@@ -419,21 +413,3 @@ void __cpu_die(unsigned int cpu)
  {
      BUG_ON("unimplemented");
  }
-
-/*
- * The following functions are defined in common/irq.c, but

common/irq.c isn't
- * built for now. These changes will be removed there when common/irq.c is
- * ready.
- */

xen/arch/riscv/stubs.c +0 -7
-
-void cf_check irq_actor_none(struct irq_desc *desc)
-{
-    BUG_ON("unimplemented");
-}
-
-unsigned int cf_check irq_startup_none(struct irq_desc *desc)
Quoted 8 lines

xen/arch/riscv/stubs.c void __cpu_die(unsigned int cpu) · line 439

Quoted 5 lines
xen/arch/riscv/stubs.c +0 -5
-{
-    BUG_ON("unimplemented");
-
-    return 0;
-}

All 3 of these are introduced in the previous patch and deleted again
here.  Looks like a rebasing accident.

Not really.

This was done to avoid build failures for RISC-V. If the HEAD is on the
previous patch where these changes are introduced and then we just drop
them, it will lead to a linkage error because these functions are defined in common/irq.c, which isn't built for RISC-V if the HEAD is on
the previous patch:
   /build/xen/arch/riscv/entry.S:86: undefined reference to `max_page'

Quoted 7 lines
Quoted 5 lines

Nothing in the series touches entry.S, so I'm not sure what this is complaining about.

The stub for get_upper_mfn_bound() references max_page, but it's only used in a SYSCTL so you can avoid the problem with a BUG_ON().

I didn't get how I can use BUG_ON() with declaration of variable to avoid the compilation issue the undefined reference to max_page?

You don't need to implement get_upper_mfn_bound() yet.

If you implement it as:

unsigned long get_upper_mfn_bound(void)
{
    BUG_ON("unimplemented");
}

in the previous patch, then you also don't have any problems with max_page between these two patches either.

Quoted 27 lines
Quoted 27 lines

BTW, you also don't need a return after a BUG_ON(). __builtin_unreachable() takes care of everything properly for you.

Quoted 22 lines

   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x8): undefined reference to
   `irq_startup_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x10): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x18): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: prelink.o:(.rodata+0x20): undefined reference to
   `irq_actor_none'
   riscv64-linux-gnu-ld: xen-syms: hidden symbol `irq_actor_none' isn't
   defined

That is why these stubs were introduced in the previous patch (because
common/irq.c isn't built at that moment) and are removed in this patch
(since at the moment of this patch, common/irq.c is now being built).

Quoted 8 lines
Quoted 6 lines

These OTOH are a side effect of how no_irq_type works, which is horrifying, and not something I can unsee.

I'll see about fixing it, because I really can't bare to leave it like
this...

I am really sorry, but I don't understand should I deal with this somehow now or the provided changes are okay?

I've emailed out a 2-patch series to unbreak this.

~Andrew

#28 lore raw

PATCH v12 8/8 xen/README: add compiler and binutils versions for RISC-V64

In reply to Oleksii Kurochko

This patch doesn't represent a strict lower bound for GCC and GNU Binutils; rather, these versions are specifically employed by the Xen RISC-V container and are anticipated to undergo continuous testing. Older GCC and GNU Binutils would work,
but this is not a guarantee.

While it is feasible to utilize Clang, it's important to note that, currently, there is no Xen RISC-V CI job in place to verify the seamless functioning of the build with Clang.

--
Changes in V5-V12:
- Nothing changed. Only rebase.
---
Changes in V6:
- update the message in README.
---
Changes in V5:
- update the commit message and README file with additional explanation about GCC and GNU Binutils version. Additionally, it was added information about Clang.
---
Changes in V4:
- Update version of GCC (12.2) and GNU Binutils (2.39) to the version which are in Xen's contrainter for RISC-V
---
Changes in V3:
- new patch

---
 README | 4 ++++
 1 file changed, 4 insertions(+)
README +4 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:
       - For ARM 64-bit:
         - GCC 5.1 or later
         - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later
+          Older GCC and GNU Binutils would work, but this is not a guarantee.
     * POSIX compatible awk
     * Development install of zlib (e.g., zlib-dev)
     * Development install of Python 2.7 or later (e.g., python-dev)
Signature · 1 line
2.45.0
#29 lore raw

Re: [PATCH v12 8/8] xen/README: add compiler and binutils versions for RISC-V64

README provided by your OS distributor: · line 54

Quoted 12 lines · Oleksii Kurochko
README +4 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:
       - For ARM 64-bit:
         - GCC 5.1 or later
         - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later
+          Older GCC and GNU Binutils would work, but this is not a guarantee.

This sentence isn't appropriate to live here.

The commit message saying "this is what we run in CI" is perfectly good enough.

With this dropped, Reviewed-by: Andrew Cooper
<[email protected]>.  Can fix on commit.

author
#30 lore raw

Re: [PATCH v12 8/8] xen/README: add compiler and binutils versions for RISC-V64

Quoted 17 lines · Andrew Cooper
Quoted 5 lines · Oleksii Kurochko
README +0 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:
Quoted 8 lines

       - For ARM 64-bit:
         - GCC 5.1 or later
         - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later
+          Older GCC and GNU Binutils would work, but this is not a guarantee.

This sentence isn't appropriate to live here.

Quoted 6 lines

The commit message saying "this is what we run in CI" is perfectly good
enough.

With this dropped, Reviewed-by: Andrew Cooper
<[email protected]>.  Can fix on commt.

I am okay with dropping this sentence, but someone ( unfortunately I don't remember who Jan? Julien? ) requested it, and I think it would be nice to hear their opinion before doing so.

~ Oleksii

#31 lore raw

Re: [PATCH v12 8/8] xen/README: add compiler and binutils versions for RISC-V64

Quoted 21 lines · Oleksii K.
Quoted 20 lines · Andrew Cooper
Quoted 5 lines · Oleksii Kurochko
README +0 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:
Quoted 8 lines

       - For ARM 64-bit:
         - GCC 5.1 or later
         - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later
+          Older GCC and GNU Binutils would work, but this is not a guarantee.

This sentence isn't appropriate to live here.

The commit message saying "this is what we run in CI" is perfectly good
enough.

Quoted 5 lines
Quoted 2 lines

With this dropped, Reviewed-by: Andrew Cooper
<[email protected]>.  Can fix on commt.

I am okay with dropping this sentence, but someone ( unfortunately I don't remember who Jan? Julien? ) requested it, and I think it would be nice to hear their opinion before doing so.

It's line noise, and literally a redundant statement.  The same is true of every other line in the file, and we don't write it because that would be incredibly stupid.

Anyone who wants to see why this was chosen can read the commit message.

~Andrew

#32 lore raw

Re: [PATCH v12 8/8] xen/README: add compiler and binutils versions for RISC-V64

Hi,

Quoted 22 lines · Andrew Cooper
Quoted 21 lines · Oleksii K.
Quoted 20 lines · Andrew Cooper
Quoted 12 lines · Oleksii Kurochko
README +4 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:
        - For ARM 64-bit:
          - GCC 5.1 or later
          - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later
+          Older GCC and GNU Binutils would work, but this is not a
Quoted 1 line

guarantee.

This sentence isn't appropriate to live here.

The commit message saying "this is what we run in CI" is perfectly good
enough.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

With this dropped, Reviewed-by: Andrew Cooper
<[email protected]>.  Can fix on commt.

I am okay with dropping this sentence, but someone ( unfortunately I don't remember who Jan? Julien? ) requested it, and I think it would be nice to hear their opinion before doing so.

I don't think it was me :). I have no issue with dropping the line.

Quoted 3 lines

It's line noise, and literally a redundant statement.  The same is true of every other line in the file,

+1. If anyone wanted extra clarify, we could spell out at the beginning of the file that this is the versions are confident with.

Cheers,

Signature · 1 line
Julien Grall
#33 lore raw

Re: [PATCH v12 8/8] xen/README: add compiler and binutils versions for RISC-V64

In reply to Oleksii Kurochko

README provided by your OS distributor: · line 53

Quoted 11 lines · Oleksii Kurochko
README +3 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:
       - For ARM 64-bit:
         - GCC 5.1 or later
         - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later

I would like to petition for GCC 10 and Binutils 2.35.

These are the versions provided as cross-compilers by Debian, and therefore are the versions I would prefer to do smoke testing with.

One issue is in cpu_relax(), needing this diff to fix:

(quoted diff) +0 -0

b/xen/arch/riscv/include/asm/processor.h
index 6846151717f7..830a05dd8e3a 100644

xen/arch/riscv/include/asm/processor.h +0 -0
@@ -67,7 +67,7 @@ static inline void cpu_relax(void)

     __asm__ __volatile__ ( "pause" ); #else
     /* Encoding of the pause instruction */ -    __asm__ __volatile__ ( ".insn 0x0100000F" ); +    __asm__ __volatile__ ( ".4byte 0x0100000F" ); #endif

     barrier();

The .insn directive appears to check that the byte pattern is a known extension, where .4byte doesn't.  AFAICT, this makes .insn pretty useless for what I'd expect is it's main purpose...

The other problem is a real issue in cmpxchg.h, already committed to staging (51dabd6312c).

RISC-V does a conditional toolchain for the Zbb extension (xen/arch/riscv/rules.mk), but unconditionally uses the ANDN instruction in emulate_xchg_1_2().

Nevertheless, this is also quite easy to fix:

(quoted diff) +0 -0

b/xen/arch/riscv/include/asm/cmpxchg.h
index d5e678c03678..12ecb0950701 100644

xen/arch/riscv/include/asm/cmpxchg.h +0 -0
@@ -18,6 +18,20 @@

         : "r" (new) \
         : "memory" );

+/*
+ * Binutils < 2.37 doesn't understand ANDN.  If the toolchain is too old, form
+ * it of a NOT+AND pair
+ */
+#ifdef __riscv_zbb
+#define ANDN_INSN(rd, rs1, rs2)                 \
+    "andn " rd ", " rs1 ", " rs2 "\n"
+#else
+#define ANDN_INSN(rd, rs1, rs2)                 \
+    "not " rd ", " rs2 "\n"                     \
+    "and " rd ", " rs1 ", " rd "\n"
+#endif
+
+
 /*
  * For LR and SC, the A extension requires that the address held in rs1 be
  * naturally aligned to the size of the operand (i.e., eight-byte aligned @@ -48,7 +62,7 @@
     \
     asm volatile ( \
         "0: lr.w" lr_sfx " %[old], %[ptr_]\n" \
-        "   andn  %[scratch], %[old], %[mask]\n" \
+        ANDN_INSN("%[scratch]", "%[old]", "%[mask]") \
         "   or   %[scratch], %[scratch], %z[new_]\n" \
         "   sc.w" sc_sfx " %[scratch], %[scratch], %[ptr_]\n" \
         "   bnez %[scratch], 0b\n" \

And with that, everything builds... but doesn't link.  I've got this:

  LDS     arch/riscv/xen.lds
riscv64-linux-gnu-ld      -T arch/riscv/xen.lds -N prelink.o \
    ./common/symbols-dummy.o -o ./.xen-syms.0
riscv64-linux-gnu-ld: prelink.o: in function `keyhandler_crash_action':
/local/xen.git/xen/common/keyhandler.c:552: undefined reference to `guest_physmap_remove_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `guest_physmap_remove_page' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value

which is completely bizarre.

keyhandler_crash_action() has no actual reference to guest_physmap_remove_page(), and keyhandler.o has no such relocation.

I'm still investigating this one.

~Andrew

#34 lore raw

Re: [PATCH v12 8/8] xen/README: add compiler and binutils versions for RISC-V64

Quoted 15 lines · Andrew Cooper

README provided by your OS distributor: · line 53

Quoted 11 lines · Oleksii Kurochko
README +3 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:
       - For ARM 64-bit:
         - GCC 5.1 or later
         - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later

I would like to petition for GCC 10 and Binutils 2.35.

Quoted 2 lines

These are the versions provided as cross-compilers by Debian, and therefore are the versions I would prefer to do smoke testing with.

See why I asked to amend the specified versions by a softening sentence that you (only now) said you dislike? The "this is what we use in CI" makes it a very random choice, entirely unrelated to the compiler's abilities.

Jan

#35 lore raw

Re: [PATCH v12 8/8] xen/README: add compiler and binutils versions for RISC-V64

Quoted 15 lines · Jan Beulich
Quoted 14 lines · Andrew Cooper

README provided by your OS distributor: · line 53

Quoted 11 lines · Oleksii Kurochko
README +3 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:
       - For ARM 64-bit:
         - GCC 5.1 or later
         - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later

I would like to petition for GCC 10 and Binutils 2.35.

Quoted 5 lines
Quoted 2 lines

These are the versions provided as cross-compilers by Debian, and therefore are the versions I would prefer to do smoke testing with.

See why I asked to amend the specified versions by a softening sentence that you (only now) said you dislike? The "this is what we use in CI" makes it a very random choice, entirely unrelated to the compiler's abilities.

"what's in CI" is an arbitrary choice, and that's *explicitly* fine and the right choice for Oleksii to have made.

He's got the hard job of making the damn thing work in the first place. Requiring him to also go and get some old compilers to backdate the support statement is unreasonable for you to demand.

In this case, I'm saying that it would be convenient for *me* if the numbers were older, because that's what *I* have and what *I'm* wanting to testing with.  This means that I'm the one taking on the responsibility of playing backwards-compatibility-roulette.

Now, for other reasons I no longer have those versions, but one of the 3 bugs I raises is still a real bug needing fixing.

~Andrew

author
#36 lore raw

Re: [PATCH v12 8/8] xen/README: add compiler and binutils versions for RISC-V64

In reply to Andrew Cooper
Quoted 15 lines · Andrew Cooper
Quoted 11 lines · Oleksii Kurochko
README +0 -0
index c8a108449e..30da5ff9c0 100644
@@ -48,6 +48,10 @@ provided by your OS distributor:

       - For ARM 64-bit:
         - GCC 5.1 or later
         - GNU Binutils 2.24 or later
+      - For RISC-V 64-bit:
+        - GCC 12.2 or later
+        - GNU Binutils 2.39 or later

I would like to petition for GCC 10 and Binutils 2.35.

Quoted 2 lines

These are the versions provided as cross-compilers by Debian, and therefore are the versions I would prefer to do smoke testing with.

We can consider that, but I prefer to make a separate patch series for that.

~ Oleksii

Quoted 95 lines

One issue is in cpu_relax(), needing this diff to fix:

xen/arch/riscv/include/asm/processor.h +1 -1
index 6846151717f7..830a05dd8e3a 100644
@@ -67,7 +67,7 @@ static inline void cpu_relax(void)
      __asm__ __volatile__ ( "pause" );
  #else
      /* Encoding of the pause instruction */
-    __asm__ __volatile__ ( ".insn 0x0100000F" );
+    __asm__ __volatile__ ( ".4byte 0x0100000F" );
  #endif


     barrier();

The .insn directive appears to check that the byte pattern is a known extension, where .4byte doesn't.  AFAICT, this makes .insn pretty useless for what I'd expect is it's main purpose...

The other problem is a real issue in cmpxchg.h, already committed to staging (51dabd6312c).

RISC-V does a conditional toolchain for the Zbb extension (xen/arch/riscv/rules.mk), but unconditionally uses the ANDN instruction
in emulate_xchg_1_2().

Nevertheless, this is also quite easy to fix:

xen/arch/riscv/include/asm/cmpxchg.h +0 -0
index d5e678c03678..12ecb0950701 100644
@@ -18,6 +18,20 @@
          : "r" (new) \
          : "memory" );


+/*
+ * Binutils < 2.37 doesn't understand ANDN.  If the toolchain is too old, form
+ * it of a NOT+AND pair
+ */

xen/arch/riscv/include/asm/cmpxchg.h +10 -0
+#ifdef __riscv_zbb
+#define ANDN_INSN(rd, rs1, rs2)                 \
+    "andn " rd ", " rs1 ", " rs2 "\n"
+#else
+#define ANDN_INSN(rd, rs1, rs2)                 \
+    "not " rd ", " rs2 "\n"                     \
+    "and " rd ", " rs1 ", " rd "\n"
+#endif
+
+

 /*
  * For LR and SC, the A extension requires that the address held in rs1 be
  * naturally aligned to the size of the operand (i.e., eight-byte aligned

xen/arch/riscv/include/asm/cmpxchg.h +1 -1
@@ -48,7 +62,7 @@
      \
      asm volatile ( \
          "0: lr.w" lr_sfx " %[old], %[ptr_]\n" \
-        "   andn  %[scratch], %[old], %[mask]\n" \
+        ANDN_INSN("%[scratch]", "%[old]", "%[mask]") \
          "   or   %[scratch], %[scratch], %z[new_]\n" \
          "   sc.w" sc_sfx " %[scratch], %[scratch], %[ptr_]\n" \
          "   bnez %[scratch], 0b\n" \

And with that, everything builds... but doesn't link.  I've got this:

  LDS     arch/riscv/xen.lds
riscv64-linux-gnu-ld      -T arch/riscv/xen.lds -N prelink.o \
    ./common/symbols-dummy.o -o ./.xen-syms.0
riscv64-linux-gnu-ld: prelink.o: in function `keyhandler_crash_action':
/local/xen.git/xen/common/keyhandler.c:552: undefined reference to `guest_physmap_remove_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `guest_physmap_remove_page' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value

which is completely bizarre.

keyhandler_crash_action() has no actual reference to guest_physmap_remove_page(), and keyhandler.o has no such relocation.

I'm still investigating this one.

~Andrew