AArch64 Linux kernel port

The patch adds the kernel booting and the initial setup code. Documentation/arm64/booting.txt describes the booting protocol on the AArch64 Linux kernel. This is subject to change following the work on boot standardisation, ACPI.

Signed-off-by Will Deacon <[email protected]>

Signed-off-by Catalin Marinas <[email protected]>

Acked-by Nicolas Pitre <[email protected]>

Acked-by Tony Lindgren <[email protected]>

The patch adds the kernel booting and the initial setup code. Documentation/arm64/booting.txt describes the booting protocol on the AArch64 Linux kernel. This is subject to change following the work on boot standardisation, ACPI.

Signed-off-by Will Deacon <[email protected]>

Signed-off-by Catalin Marinas <[email protected]>

Acked-by Nicolas Pitre <[email protected]>

Acked-by Tony Lindgren <[email protected]>

---
 Documentation/arm64/booting.txt |  152 ++++++++++++
 arch/arm64/include/asm/setup.h  |   26 ++
 arch/arm64/kernel/head.S        |  510 +++++++++++++++++++++++++++++++++++++++
 arch/arm64/kernel/setup.c       |  351 +++++++++++++++++++++++++++
 4 files changed, 1039 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/arm64/booting.txt
 create mode 100644 arch/arm64/include/asm/setup.h
 create mode 100644 arch/arm64/kernel/head.S
 create mode 100644 arch/arm64/kernel/setup.c

Documentation/arm64/booting.txt new file +41 -0

new file mode 100644
index 0000000..9c4d388
@@ -0,0 +1,152 @@
+			Booting AArch64 Linux
+			=====================
+
+Author: Will Deacon <[email protected]>
+Date  : 07 September 2012
+
+This document is based on the ARM booting document by Russell King and
+is relevant to all public releases of the AArch64 Linux kernel.
+
+The AArch64 exception model is made up of a number of exception levels
+(EL0 - EL3), with EL0 and EL1 having a secure and a non-secure
+counterpart.  EL2 is the hypervisor level and exists only in non-secure
+mode. EL3 is the highest priority level and exists only in secure mode.
+
+For the purposes of this document, we will use the term `boot loader'
+simply to define all software that executes on the CPU(s) before control
+is passed to the Linux kernel.  This may include secure monitor and
+hypervisor code, or it may just be a handful of instructions for
+preparing a minimal boot environment.
+
+Essentially, the boot loader should provide (as a minimum) the
+following:
+
+1. Setup and initialise the RAM
+2. Setup the device tree
+3. Decompress the kernel image
+4. Call the kernel image
+
+
+1. Setup and initialise RAM
+---------------------------
+
+Requirement: MANDATORY
+
+The boot loader is expected to find and initialise all RAM that the
+kernel will use for volatile data storage in the system.  It performs
+this in a machine dependent manner.  (It may use internal algorithms
+to automatically locate and size all RAM, or it may use knowledge of
+the RAM in the machine, or any other method the boot loader designer
+sees fit.)
+

Documentation/arm64/booting.txt new file +16 -0

+
+
+3. Decompress the kernel image
+------------------------------
+
+Requirement: OPTIONAL
+
+The AArch64 kernel does not currently provide a decompressor and
+therefore requires decompression (gzip etc.) to be performed by the boot
+loader if a compressed Image target (e.g. Image.gz) is used.  For
+bootloaders that do not implement this requirement, the uncompressed
+Image target is available instead.
+
+
+4. Call the kernel image
+------------------------

Documentation/arm64/booting.txt new file +19 -0

+
+Before jumping into the kernel, the following conditions must be met:
+
+- Quiesce all DMA capable devices so that memory does not get
+  corrupted by bogus network packets or disk data.  This will save
+  you many hours of debug.
+
+- Primary CPU general-purpose register settings
+  x0 = physical address of device tree blob (dtb) in system RAM.
+  x1 = 0 (reserved for future use)
+  x2 = 0 (reserved for future use)
+  x3 = 0 (reserved for future use)
+
+- CPU mode
+  All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
+  IRQ and FIQ).
+  The CPU must be in either EL2 (RECOMMENDED in order to have access to
+  the virtualisation extensions) or non-secure EL1.
+

Documentation/arm64/booting.txt new file +26 -0

+
+- Coherency
+  All CPUs to be booted by the kernel must be part of the same coherency
+  domain on entry to the kernel.  This may require IMPLEMENTATION DEFINED
+  initialisation to enable the receiving of maintenance operations on
+  each CPU.
+
+- System registers
+  All writable architected system registers at the exception level where
+  the kernel image will be entered must be initialised by software at a
+  higher exception level to prevent execution in an UNKNOWN state.
+
+The boot loader is expected to enter the kernel on each CPU in the
+following manner:
+
+- The primary CPU must jump directly to the first instruction of the
+  kernel image.  The device tree blob passed by this CPU must contain
+  for each CPU node:
+
+    1. An 'enable-method' property. Currently, the only supported value
+       for this field is the string "spin-table".
+
+    2. A 'cpu-release-addr' property identifying a 64-bit,
+       zero-initialised memory location.
+
+  It is expected that the bootloader will generate these device tree

Documentation/arm64/booting.txt line 77

Quoted 12 lines · Catalin Marinas

+4. Call the kernel image
+------------------------

Documentation/arm64/booting.txt new file +10 -0

+
+Requirement: MANDATORY
+
+The decompressed kernel image contains a 32-byte header as follows:
+
+  u32 magic	= 0x14000008;	/* branch to stext, little-endian */
+  u32 res0	= 0;		/* reserved */
+  u64 text_offset;		/* Image load offset */
+  u64 res1	= 0;		/* reserved */
+  u64 res2	= 0;		/* reserved */

we need to have a magic to known it's a arm64 kernel

Documentation/arm64/booting.txt line 81

Quoted 4 lines

Documentation/arm64/booting.txt new file +4 -0

+
+The image must be placed at the specified offset (currently 0x80000)
+from the start of the system RAM and called there. The start of the
+system RAM must be aligned to 2MB.

can we drop this

Quoted 14 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 77

Quoted 12 lines · Catalin Marinas

+4. Call the kernel image
+------------------------

Documentation/arm64/booting.txt new file +10 -0

+
+Requirement: MANDATORY
+
+The decompressed kernel image contains a 32-byte header as follows:
+
+  u32 magic	= 0x14000008;	/* branch to stext, little-endian */
+  u32 res0	= 0;		/* reserved */
+  u64 text_offset;		/* Image load offset */
+  u64 res1	= 0;		/* reserved */
+  u64 res2	= 0;		/* reserved */

we need to have a magic to known it's a arm64 kernel

You have it: it's 0x14000008 at the beginning.

Quoted 6 lines

Documentation/arm64/booting.txt line 81

Quoted 5 lines

Documentation/arm64/booting.txt new file +4 -0

+
+The image must be placed at the specified offset (currently 0x80000)
+from the start of the system RAM and called there. The start of the
+system RAM must be aligned to 2MB.

can we drop this

Drop what?
And why?

Quoted 9 lines · Nicolas Pitre

Quoted 5 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 81

Quoted 3 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +3 -0

+The image must be placed at the specified offset (currently 0x80000)
+from the start of the system RAM and called there. The start of the
+system RAM must be aligned to 2MB.

can we drop this

Drop what?
And why?

This contrain the must be loadable at any address

Quoted 8 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 7 lines · Nicolas Pitre

Quoted 5 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 81

Quoted 3 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +3 -0

+The image must be placed at the specified offset (currently 0x80000)
+from the start of the system RAM and called there. The start of the
+system RAM must be aligned to 2MB.

can we drop this

Quoted 3 lines

Quoted 2 lines

Drop what?
And why?

This contrain the must be loadable at any address

You can't easily load the kernel image at any address, unless it can relocate itself and you have a way to specify PHYS_OFFSET. We don't want a compile-time PHYS_OFFSET, the kernel detects it at boot time based on the load address.

Quoted 13 lines · Catalin Marinas

Quoted 11 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 9 lines · Nicolas Pitre

Quoted 5 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 81

Quoted 3 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +3 -0

+The image must be placed at the specified offset (currently 0x80000)
+from the start of the system RAM and called there. The start of the
+system RAM must be aligned to 2MB.

can we drop this

Drop what?
And why?

This contrain the must be loadable at any address

Quoted 4 lines

You can't easily load the kernel image at any address, unless it can relocate itself and you have a way to specify PHYS_OFFSET. We don't want a compile-time PHYS_OFFSET, the kernel detects it at boot time based on the load address.

I think a bunch of other architectures can have relocatable kernels, which is useful e.g. for kdump. It does imply a small runtime cost and may have other disadvantages though.

Quoted 5 lines · Arnd Bergmann

Quoted 1 line · Catalin Marinas

On Mon, Sep 10, 2012 at 06:53:39AM +0100, Jean-Christophe PLAGNIOL-VILLARD wrote:

I think a bunch of other architectures can have relocatable kernels, which is useful e.g. for kdump. It does imply a small runtime cost and may have other disadvantages though.

Relocatable in physical space is what kdump actually needs, and that's what we already have here (as well as on ARM32 for that matter with CONFIG_ARM_PATCH_PHYS_VIRT). Relocatable in the virtual space is costly and we shouldn't need to go there.

Quoted 7 lines · Nicolas Pitre

Quoted 5 lines · Arnd Bergmann

Quoted 1 line · Catalin Marinas

On Mon, Sep 10, 2012 at 06:53:39AM +0100, Jean-Christophe PLAGNIOL-VILLARD wrote:

I think a bunch of other architectures can have relocatable kernels, which is useful e.g. for kdump. It does imply a small runtime cost and may have other disadvantages though.

Quoted 4 lines

Relocatable in physical space is what kdump actually needs, and that's what we already have here (as well as on ARM32 for that matter with CONFIG_ARM_PATCH_PHYS_VIRT). Relocatable in the virtual space is costly and we shouldn't need to go there.

Ah, I see. I thought that the other architectures (powerpc and x86) doing this were actually building with -fPIC, but they do the same kind of early patching that we do.

Quoted 7 lines · Nicolas Pitre

Quoted 5 lines · Arnd Bergmann

Quoted 1 line · Catalin Marinas

On Mon, Sep 10, 2012 at 06:53:39AM +0100, Jean-Christophe PLAGNIOL-VILLARD wrote:

I think a bunch of other architectures can have relocatable kernels, which is useful e.g. for kdump. It does imply a small runtime cost and may have other disadvantages though.

Quoted 4 lines

Relocatable in physical space is what kdump actually needs, and that's what we already have here (as well as on ARM32 for that matter with CONFIG_ARM_PATCH_PHYS_VIRT). Relocatable in the virtual space is costly and we shouldn't need to go there.

Ah, I see. I thought that the other architectures (powerpc and x86) doing this were actually building with -fPIC, but they do the same kind of early patching that we do.

Quoted 8 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 7 lines · Nicolas Pitre

Quoted 5 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 81

Quoted 3 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +3 -0

+The image must be placed at the specified offset (currently 0x80000)
+from the start of the system RAM and called there. The start of the
+system RAM must be aligned to 2MB.

can we drop this

Quoted 3 lines

Quoted 2 lines

Drop what?
And why?

This contrain the must be loadable at any address

You can't easily load the kernel image at any address, unless it can relocate itself and you have a way to specify PHYS_OFFSET. We don't want a compile-time PHYS_OFFSET, the kernel detects it at boot time based on the load address.

Quoted 11 lines · Catalin Marinas

Quoted 10 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 9 lines · Nicolas Pitre

Quoted 5 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 81

Quoted 3 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +3 -0

+The image must be placed at the specified offset (currently 0x80000)
+from the start of the system RAM and called there. The start of the
+system RAM must be aligned to 2MB.

can we drop this

Drop what?
And why?

Quoted 6 lines

Quoted 1 line

This contrain the must be loadable at any address

You can't easily load the kernel image at any address, unless it can relocate itself and you have a way to specify PHYS_OFFSET. We don't want a compile-time PHYS_OFFSET, the kernel detects it at boot time based on the load address.

so NACK kexec and other boot loaders require it

Quoted 11 lines · Catalin Marinas

Quoted 10 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 9 lines · Nicolas Pitre

Quoted 5 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 81

Quoted 3 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +3 -0

+The image must be placed at the specified offset (currently 0x80000)
+from the start of the system RAM and called there. The start of the
+system RAM must be aligned to 2MB.

can we drop this

Drop what?
And why?

Quoted 6 lines

Quoted 1 line

This contrain the must be loadable at any address

You can't easily load the kernel image at any address, unless it can relocate itself and you have a way to specify PHYS_OFFSET. We don't want a compile-time PHYS_OFFSET, the kernel detects it at boot time based on the load address.

so NACK kexec and other boot loaders require it

Quoted 17 lines · Nicolas Pitre

Quoted 14 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 77

Quoted 12 lines · Catalin Marinas

+4. Call the kernel image
+------------------------

Documentation/arm64/booting.txt new file +10 -0

+
+Requirement: MANDATORY
+
+The decompressed kernel image contains a 32-byte header as follows:
+
+  u32 magic	= 0x14000008;	/* branch to stext, little-endian */
+  u32 res0	= 0;		/* reserved */
+  u64 text_offset;		/* Image load offset */
+  u64 res1	= 0;		/* reserved */
+  u64 res2	= 0;		/* reserved */

we need to have a magic to known it's a arm64 kernel

Quoted 1 line

You have it: it's 0x14000008 at the beginning.

fragile

Documentation/arm64/booting.txt line 51

Quoted 4 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +4 -0

+The device tree blob (dtb) must be no bigger than 2 megabytes in size
+and placed at a 2-megabyte boundary within the first 512 megabytes from
+the start of the kernel image. This is to allow the kernel to map the
+blob using a single section mapping in the initial page tables.

why do you want to restrict the DT to be less tahn 2MiB?

Quoted 21 lines

Documentation/arm64/booting.txt new file +21 -0

+Before jumping into the kernel, the following conditions must be met:
+
+- Quiesce all DMA capable devices so that memory does not get
+  corrupted by bogus network packets or disk data.  This will save
+  you many hours of debug.
+
+- Primary CPU general-purpose register settings
+  x0 = physical address of device tree blob (dtb) in system RAM.
+  x1 = 0 (reserved for future use)
+  x2 = 0 (reserved for future use)
+  x3 = 0 (reserved for future use)
+
+- CPU mode
+  All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
+  IRQ and FIQ).
+  The CPU must be in either EL2 (RECOMMENDED in order to have access to
+  the virtualisation extensions) or non-secure EL1.
+
+- Caches, MMUs
+  The MMU must be off.
+  Instruction cache may be on or off.

Documentation/arm64/booting.txt line 110

Quoted 7 lines

Documentation/arm64/booting.txt new file +7 -0

+  Data cache must be off and invalidated.
+  External caches (if present) must be configured and disabled.
+
+- Architected timers
+  CNTFRQ must be programmed with the timer frequency.
+  If entering the kernel at EL1, CNTHCTL_EL2 must have EL1PCTEN (bit 0)
+  set where available.

can you explain why?

Quoted 14 lines

+- The primary CPU must jump directly to the first instruction of the

Documentation/arm64/booting.txt new file +13 -0

+  kernel image.  The device tree blob passed by this CPU must contain
+  for each CPU node:
+
+    1. An 'enable-method' property. Currently, the only supported value
+       for this field is the string "spin-table".
+
+    2. A 'cpu-release-addr' property identifying a 64-bit,
+       zero-initialised memory location.
+
+  It is expected that the bootloader will generate these device tree
+  properties and insert them into the blob prior to kernel entry.
+
+- Any secondary CPUs must spin outside of the kernel in a reserved area

Documentation/arm64/booting.txt line 146

Quoted 7 lines

Documentation/arm64/booting.txt new file +7 -0

+  of memory (communicated to the kernel by a /memreserve/ region in the
+  device tree) polling their cpu-release-addr location, which must be
+  contained in the reserved region.  A wfe instruction may be inserted
+  to reduce the overhead of the busy-loop and a sev will be issued by
+  the primary CPU.  When a read of the location pointed to by the
+  cpu-release-addr returns a non-zero value, the CPU must jump directly
+  to this value.

do you plan AMP boot?

Quoted 6 lines · Jean-Christophe PLAGNIOL-VILLARD

Documentation/arm64/booting.txt line 51

Quoted 4 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +4 -0

+The device tree blob (dtb) must be no bigger than 2 megabytes in size
+and placed at a 2-megabyte boundary within the first 512 megabytes from
+the start of the kernel image. This is to allow the kernel to map the
+blob using a single section mapping in the initial page tables.

why do you want to restrict the DT to be less tahn 2MiB?

That's a restriction due on the initial memory map. At some point we may add support in head.S to parse the dtb and extract the size information. Not critical at this stage.

Quoted 21 lines

Quoted 21 lines

Documentation/arm64/booting.txt new file +21 -0

+Before jumping into the kernel, the following conditions must be met:
+
+- Quiesce all DMA capable devices so that memory does not get
+  corrupted by bogus network packets or disk data.  This will save
+  you many hours of debug.
+
+- Primary CPU general-purpose register settings
+  x0 = physical address of device tree blob (dtb) in system RAM.
+  x1 = 0 (reserved for future use)
+  x2 = 0 (reserved for future use)
+  x3 = 0 (reserved for future use)
+
+- CPU mode
+  All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
+  IRQ and FIQ).
+  The CPU must be in either EL2 (RECOMMENDED in order to have access to
+  the virtualisation extensions) or non-secure EL1.
+
+- Caches, MMUs
+  The MMU must be off.
+  Instruction cache may be on or off.

Quoted 8 lines

Documentation/arm64/booting.txt line 110

Quoted 7 lines

Documentation/arm64/booting.txt new file +7 -0

+  Data cache must be off and invalidated.
+  External caches (if present) must be configured and disabled.
+
+- Architected timers
+  CNTFRQ must be programmed with the timer frequency.
+  If entering the kernel at EL1, CNTHCTL_EL2 must have EL1PCTEN (bit 0)
+  set where available.

can you explain why?

Otherwise the kernel cannot access the generic timer registers (it is described in the AArch64 exception model which isn't public yet).

Quoted 15 lines

Quoted 15 lines

+- The primary CPU must jump directly to the first instruction of the

Documentation/arm64/booting.txt new file +13 -0

+  kernel image.  The device tree blob passed by this CPU must contain
+  for each CPU node:
+
+    1. An 'enable-method' property. Currently, the only supported value
+       for this field is the string "spin-table".
+
+    2. A 'cpu-release-addr' property identifying a 64-bit,
+       zero-initialised memory location.
+
+  It is expected that the bootloader will generate these device tree
+  properties and insert them into the blob prior to kernel entry.
+
+- Any secondary CPUs must spin outside of the kernel in a reserved area

Quoted 8 lines

Documentation/arm64/booting.txt line 146

Quoted 7 lines

Documentation/arm64/booting.txt new file +7 -0

+  of memory (communicated to the kernel by a /memreserve/ region in the
+  device tree) polling their cpu-release-addr location, which must be
+  contained in the reserved region.  A wfe instruction may be inserted
+  to reduce the overhead of the busy-loop and a sev will be issued by
+  the primary CPU.  When a read of the location pointed to by the
+  cpu-release-addr returns a non-zero value, the CPU must jump directly
+  to this value.

do you plan AMP boot?

What do you mean by AMP?

If you only want to use 3 CPUs out of 4 for example, you change the FDT information that gets passed to the kernel accordingly. So the kernel wouldn't touch the 4th one.

Quoted 32 lines · Catalin Marinas

Quoted 30 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 18 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +18 -0

+Before jumping into the kernel, the following conditions must be met:
+
+- Quiesce all DMA capable devices so that memory does not get
+  corrupted by bogus network packets or disk data.  This will save
+  you many hours of debug.
+
+- Primary CPU general-purpose register settings
+  x0 = physical address of device tree blob (dtb) in system RAM.
+  x1 = 0 (reserved for future use)
+  x2 = 0 (reserved for future use)
+  x3 = 0 (reserved for future use)
+
+- CPU mode
+  All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
+  IRQ and FIQ).
+  The CPU must be in either EL2 (RECOMMENDED in order to have access to
+  the virtualisation extensions) or non-secure EL1.
+

Documentation/arm64/booting.txt line 110

Quoted 10 lines

Documentation/arm64/booting.txt new file +10 -0

+- Caches, MMUs
+  The MMU must be off.
+  Instruction cache may be on or off.
+  Data cache must be off and invalidated.
+  External caches (if present) must be configured and disabled.
+
+- Architected timers
+  CNTFRQ must be programmed with the timer frequency.
+  If entering the kernel at EL1, CNTHCTL_EL2 must have EL1PCTEN (bit 0)
+  set where available.

can you explain why?

Quoted 2 lines

Otherwise the kernel cannot access the generic timer registers (it is described in the AArch64 exception model which isn't public yet).

I do not like the idea to do too much in the boot loader

can we drop it and do it the head.S or find an other way

Quoted 23 lines

Quoted 22 lines

Quoted 11 lines

+- The primary CPU must jump directly to the first instruction of the

Documentation/arm64/booting.txt new file +10 -0

+  kernel image.  The device tree blob passed by this CPU must contain
+  for each CPU node:
+
+    1. An 'enable-method' property. Currently, the only supported value
+       for this field is the string "spin-table".
+
+    2. A 'cpu-release-addr' property identifying a 64-bit,
+       zero-initialised memory location.
+
+  It is expected that the bootloader will generate these device tree

Documentation/arm64/booting.txt line 146

Quoted 10 lines

Documentation/arm64/booting.txt new file +10 -0

+  properties and insert them into the blob prior to kernel entry.
+
+- Any secondary CPUs must spin outside of the kernel in a reserved area
+  of memory (communicated to the kernel by a /memreserve/ region in the
+  device tree) polling their cpu-release-addr location, which must be
+  contained in the reserved region.  A wfe instruction may be inserted
+  to reduce the overhead of the busy-loop and a sev will be issued by
+  the primary CPU.  When a read of the location pointed to by the
+  cpu-release-addr returns a non-zero value, the CPU must jump directly
+  to this value.

do you plan AMP boot?

Quoted 5 lines

What do you mean by AMP?

If you only want to use 3 CPUs out of 4 for example, you change the FDT information that gets passed to the kernel accordingly. So the kernel wouldn't touch the 4th one.

I mean boot a different kernel in each core as example

Quoted 33 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 32 lines · Catalin Marinas

Quoted 30 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 18 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +18 -0

+Before jumping into the kernel, the following conditions must be met:
+
+- Quiesce all DMA capable devices so that memory does not get
+  corrupted by bogus network packets or disk data.  This will save
+  you many hours of debug.
+
+- Primary CPU general-purpose register settings
+  x0 = physical address of device tree blob (dtb) in system RAM.
+  x1 = 0 (reserved for future use)
+  x2 = 0 (reserved for future use)
+  x3 = 0 (reserved for future use)
+
+- CPU mode
+  All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
+  IRQ and FIQ).
+  The CPU must be in either EL2 (RECOMMENDED in order to have access to
+  the virtualisation extensions) or non-secure EL1.
+

Documentation/arm64/booting.txt line 110

Quoted 10 lines

Documentation/arm64/booting.txt new file +10 -0

+- Caches, MMUs
+  The MMU must be off.
+  Instruction cache may be on or off.
+  Data cache must be off and invalidated.
+  External caches (if present) must be configured and disabled.
+
+- Architected timers
+  CNTFRQ must be programmed with the timer frequency.
+  If entering the kernel at EL1, CNTHCTL_EL2 must have EL1PCTEN (bit 0)
+  set where available.

can you explain why?

Quoted 3 lines

Quoted 2 lines

Otherwise the kernel cannot access the generic timer registers (it is described in the AArch64 exception model which isn't public yet).

I do not like the idea to do too much in the boot loader

git://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/boot-wrapper-aarch64.git

Quoted 36 lines · Catalin Marinas

Quoted 35 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 34 lines · Catalin Marinas

Quoted 22 lines · Jean-Christophe PLAGNIOL-VILLARD

Quoted 21 lines · Catalin Marinas

Documentation/arm64/booting.txt new file +21 -0

+Before jumping into the kernel, the following conditions must be met:
+
+- Quiesce all DMA capable devices so that memory does not get
+  corrupted by bogus network packets or disk data.  This will save
+  you many hours of debug.
+
+- Primary CPU general-purpose register settings
+  x0 = physical address of device tree blob (dtb) in system RAM.
+  x1 = 0 (reserved for future use)
+  x2 = 0 (reserved for future use)
+  x3 = 0 (reserved for future use)
+
+- CPU mode
+  All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,
+  IRQ and FIQ).
+  The CPU must be in either EL2 (RECOMMENDED in order to have access to
+  the virtualisation extensions) or non-secure EL1.
+
+- Caches, MMUs
+  The MMU must be off.
+  Instruction cache may be on or off.

Quoted 8 lines

Documentation/arm64/booting.txt line 110

Quoted 7 lines

Documentation/arm64/booting.txt new file +7 -0

+  Data cache must be off and invalidated.
+  External caches (if present) must be configured and disabled.
+
+- Architected timers
+  CNTFRQ must be programmed with the timer frequency.
+  If entering the kernel at EL1, CNTHCTL_EL2 must have EL1PCTEN (bit 0)
+  set where available.

can you explain why?

Otherwise the kernel cannot access the generic timer registers (it is described in the AArch64 exception model which isn't public yet).

Quoted 1 line

Quoted 1 line

I do not like the idea to do too much in the boot loader

I agree that where possible we should minimize the amount of work required of bootloaders. I'm not sure that the percentage of work we might be able to trim away on hardware would be significant, but on simulators I think we might be able to get to the point where, if you can specify the r0 reset value to be the dtb pointer, you can just load a devicetree blob and kernel image into memory and start executing the kernel, no boot wrapper required.

I don't know if Jean-Christophe was commenting on just the EL1PCTEN bit or

also the CNTFRQ register, but with regard to the latter, it appears to me that the timer code actually reads the frequency primarily from the devicetree and only reads CNTFRQ if the "clock-frequency" property is unspecified or specified as zero.

I suggest that the kernel calling requirements be modified to communicate this fallback, rather than mandatory, status, unless some not-yet-released part of the arm64 kernel needs it. If userspace code will need to use CNTFRQ at some point then perhaps that should be mentioned in an informational note rather than being presented as a kernel booting requirement.

Quoted 1 line

git://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/boot-wrapper-aarch64.git

The UART and GIC stuff is board-specific, and I don't currently have any suggestions about the multicore initialization, but would you object to the kernel eventually being equipped to make the descent from EL3 itself?

The patch contains the exception entry code (kernel/entry.S), pt_regs structure and related accessors, undefined instruction trapping and stack tracing.

AArch64 Linux kernel (including kernel threads) runs in EL1 mode using the SP1 stack. The vectors don't have a fixed address, only alignment (2^11) requirements.

Signed-off-by Will Deacon <[email protected]>

Signed-off-by Catalin Marinas <[email protected]>

Acked-by Tony Lindgren <[email protected]>

The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration.

PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage).

On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables.

The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default.

Signed-off-by Will Deacon <[email protected]>

Signed-off-by Catalin Marinas <[email protected]>

Acked-by Tony Lindgren <[email protected]>

This patch contains the initialisation of the memory blocks, MMU attributes and the memory map. Only five memory types are defined:
Device nGnRnE (equivalent to Strongly Ordered), Device nGnRE (classic Device memory), Device GRE, Normal Non-cacheable and Normal Cacheable. Cache policies are supported via the memory attributes register (MAIR_EL1) and only affect the Normal Cacheable mappings.

This patch also adds the SPARSEMEM_VMEMMAP initialisation.

Signed-off-by Will Deacon <[email protected]>

Signed-off-by Catalin Marinas <[email protected]>

Acked-by Tony Lindgren <[email protected]>

This patch adds support for the handling of the MMU faults (exception entry code introduced by a previous patch) and page table management.

The user translation table is pointed to by TTBR0 and the kernel one (swapper_pg_dir) by TTBR1. There is no translation information shared or address space overlapping between user and kernel page tables.

Signed-off-by Will Deacon <[email protected]>

Signed-off-by Catalin Marinas <[email protected]>

Acked-by Tony Lindgren <[email protected]>

The patch adds support for thread creation and context switching. The context switching CPU specific code is introduced with the CPU support patch (part of the arch/arm64/mm/proc.S file). AArch64 supports ASID-tagged TLBs and the ASID can be either 8 or 16-bit wide (detectable via the ID_AA64AFR0_EL1 register).

Signed-off-by Will Deacon <[email protected]>

Signed-off-by Catalin Marinas <[email protected]>

Acked-by Tony Lindgren <[email protected]>

arch/arm64/kernel/cputable.c new file +14 -0

+
+#include <asm/cputable.h>
+
+extern unsigned long __cpu_setup(void);
+
+struct cpu_info __initdata cpu_table[] = {
+	{
+		.cpu_id_val	= 0x000f0000,
+		.cpu_id_mask	= 0x000f0000,
+		.cpu_name	= "AArch64 Processor",
+		.cpu_setup	= __cpu_setup,
+	},
+	{ /* Empty */ },
+};

Quoted 13 lines · Catalin Marinas

arch/arm64/kernel/cputable.c new file +13 -0

+#include <asm/cputable.h>
+
+extern unsigned long __cpu_setup(void);
+
+struct cpu_info __initdata cpu_table[] = {
+	{
+		.cpu_id_val	= 0x000f0000,
+		.cpu_id_mask	= 0x000f0000,
+		.cpu_name	= "AArch64 Processor",
+		.cpu_setup	= __cpu_setup,
+	},
+	{ /* Empty */ },
+};

arch/arm64/mm/proc.S line 47

Quoted 9 lines

arch/arm64/mm/proc.S new file +9 -0

+#define MAIR(attr, mt)	((attr) << ((mt) * 8))
+
+ENTRY(cpu_proc_init)
+	ret
+ENDPROC(cpu_proc_init)
+
+ENTRY(cpu_proc_fin)
+	ret
+ENDPROC(cpu_proc_fin)

These seem to be leftovers from multi-cpu support that can be removed.

+ * Invalidate the data cache within the specified region; we will be
+ * performing a DMA operation in this region and we want to purge old
+ * data in the cache.
+ *
+ * - start - virtual start address of region
+ * - end - virtual end address of region
+ */

arch/arm64/mm/cache.S new file +62 -0

+ENTRY(dmac_inv_range)
+	dcache_line_size x2, x3
+	sub	x3, x2, #1
+	bic	x0, x0, x3
+	bic	x1, x1, x3
+1:	dc	ivac, x0			// invalidate D / U line
+	add	x0, x0, x2
+	cmp	x0, x1
+	b.lo	1b
+	dsb	sy
+	ret
+ENDPROC(dmac_inv_range)
+
+/*
+ *	dmac_clean_range(start,end)
+ *	- start   - virtual start address of region
+ *	- end     - virtual end address of region
+ */
+ENTRY(dmac_clean_range)
+	dcache_line_size x2, x3
+	sub	x3, x2, #1
+	bic	x0, x0, x3
+1:	dc	cvac, x0			// clean D / U line
+	add	x0, x0, x2
+	cmp	x0, x1
+	b.lo	1b
+	dsb	sy
+	ret
+ENDPROC(dmac_clean_range)
+
+/*
+ *	dmac_flush_range(start,end)
+ *	- start   - virtual start address of region
+ *	- end     - virtual end address of region
+ */
+ENTRY(dmac_flush_range)
+	dcache_line_size x2, x3
+	sub	x3, x2, #1
+	bic	x0, x0, x3
+1:	dc	civac, x0			// clean & invalidate D / U line
+	add	x0, x0, x2
+	cmp	x0, x1
+	b.lo	1b
+	dsb	sy
+	ret
+ENDPROC(dmac_flush_range)
+
+/*
+ *	dmac_map_area(start, size, dir)
+ *	- start	- kernel virtual start address
+ *	- size	- size of region
+ *	- dir	- DMA direction
+ */
+ENTRY(dmac_map_area)
+	add	x1, x1, x0
+	cmp	x2, #DMA_FROM_DEVICE
+	b.eq	dmac_inv_range
+	b	dmac_clean_range
+ENDPROC(dmac_map_area)
+
+/*
+ *	dmac_unmap_area(start, size, dir)

arch/arm64/mm/cache.S line 171

Quoted 3 lines · Catalin Marinas

arch/arm64/mm/cache.S new file +3 -0

+
+/*
+ *	dmac_inv_range(start,end)

all of these appear to be unused now. Can you remove them?

Quoted 6 lines · Arnd Bergmann

arch/arm64/mm/cache.S line 171

Quoted 3 lines · Catalin Marinas

arch/arm64/mm/cache.S new file +3 -0

+
+/*
+ *	dmac_inv_range(start,end)

all of these appear to be unused now. Can you remove them?

They aren't currently used but I expect some partners to make use of them on their private development. I could drop them from the upstream branch temporarily.

Quoted 8 lines · Catalin Marinas

Quoted 6 lines · Arnd Bergmann

arch/arm64/mm/cache.S line 171

Quoted 3 lines · Catalin Marinas

arch/arm64/mm/cache.S new file +3 -0

+
+/*
+ *	dmac_inv_range(start,end)

all of these appear to be unused now. Can you remove them?

Quoted 3 lines

They aren't currently used but I expect some partners to make use of them on their private development. I could drop them from the upstream branch temporarily.

Depends. If your expectation from those partners is a real probability then it is best to keep a skeleton in place for them to use, otherwise they might get ... creative. If this is just wild speculation then you should drop it.

The patch adds functionality required for cache maintenance. The AArch64 architecture mandates non-aliasing VIPT or PIPT D-cache and VIPT (may have aliases) or ASID-tagged VIVT I-cache. Cache maintenance operations are automatically broadcast in hardware between CPUs.

Signed-off-by Will Deacon <[email protected]>

Signed-off-by Catalin Marinas <[email protected]>

Acked-by Tony Lindgren <[email protected]>

---
 arch/arm64/include/asm/cache.h      |   32 +++++
 arch/arm64/include/asm/cacheflush.h |  168 +++++++++++++++++++++++
 arch/arm64/include/asm/cachetype.h  |   48 +++++++
 arch/arm64/mm/cache.S               |  251 +++++++++++++++++++++++++++++++++++
 arch/arm64/mm/flush.c               |  138 +++++++++++++++++++
 5 files changed, 637 insertions(+), 0 deletions(-)
 create mode 100644 arch/arm64/include/asm/cache.h
 create mode 100644 arch/arm64/include/asm/cacheflush.h
 create mode 100644 arch/arm64/include/asm/cachetype.h
 create mode 100644 arch/arm64/mm/cache.S
 create mode 100644 arch/arm64/mm/flush.c

arch/arm64/include/asm/cache.h new file +32 -0

new file mode 100644
index 0000000..390308a
@@ -0,0 +1,32 @@
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ASM_CACHE_H
+#define __ASM_CACHE_H
+
+#define L1_CACHE_SHIFT		6
+#define L1_CACHE_BYTES		(1 << L1_CACHE_SHIFT)
+
+/*
+ * Memory returned by kmalloc() may be used for DMA, so we must make
+ * sure that all such allocations are cache aligned. Otherwise,
+ * unrelated code may cause parts of the buffer to be read into the
+ * cache before the transfer is done, causing old data to be seen by
+ * the CPU.
+ */
+#define ARCH_DMA_MINALIGN	L1_CACHE_BYTES
+#define ARCH_SLAB_MINALIGN	8
+
+#endif

arch/arm64/include/asm/cacheflush.h new file +121 -0

new file mode 100644
index 0000000..0bb9853
@@ -0,0 +1,168 @@
+/*
+ * Based on arch/arm/include/asm/cacheflush.h
+ *
+ * Copyright (C) 1999-2002 Russell King.
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ASM_CACHEFLUSH_H
+#define __ASM_CACHEFLUSH_H
+
+#include <linux/mm.h>
+
+/*
+ * This flag is used to indicate that the page pointed to by a pte is clean
+ * and does not require cleaning before returning it to the user.
+ */
+#define PG_dcache_clean PG_arch_1
+
+/*
+ *	MM Cache Management
+ *	===================
+ *
+ *	The arch/arm64/mm/cache.S implements these methods.
+ *
+ *	Start addresses are inclusive and end addresses are exclusive; start
+ *	addresses should be rounded down, end addresses up.
+ *
+ *	See Documentation/cachetlb.txt for more information. Please note that
+ *	the implementation assumes non-aliasing VIPT D-cache and (aliasing)
+ *	VIPT or ASID-tagged VIVT I-cache.
+ *
+ *	flush_cache_all()
+ *
+ *		Unconditionally clean and invalidate the entire cache.
+ *
+ *	flush_cache_mm(mm)
+ *
+ *		Clean and invalidate all user space cache entries
+ *		before a change of page tables.
+ *
+ *	flush_icache_range(start, end)
+ *
+ *		Ensure coherency between the I-cache and the D-cache in the
+ *		region described by start, end.
+ *		- start  - virtual start address
+ *		- end    - virtual end address
+ *
+ *	__flush_cache_user_range(start, end)
+ *
+ *		Ensure coherency between the I-cache and the D-cache in the
+ *		region described by start, end.
+ *		- start  - virtual start address
+ *		- end    - virtual end address
+ *
+ *	__flush_dcache_area(kaddr, size)
+ *
+ *		Ensure that the data held in page is written back.
+ *		- kaddr  - page address
+ *		- size   - region size
+ */
+extern void flush_cache_all(void);
+extern void flush_cache_mm(struct mm_struct *mm);
+extern void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end);
+extern void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn);
+extern void flush_icache_range(unsigned long start, unsigned long end);
+extern void __flush_dcache_area(void *addr, size_t len);
+extern void __flush_cache_user_range(unsigned long start, unsigned long end);
+
+/*
+ * These are private to the dma-mapping API.  Do not use directly.
+ * Their sole purpose is to ensure that data held in the cache
+ * is visible to DMA, or data written by DMA to system memory is
+ * visible to the CPU.
+ */
+extern void dmac_map_area(const void *, size_t, int);
+extern void dmac_unmap_area(const void *, size_t, int);
+extern void dmac_flush_range(const void *, const void *);
+
+/*
+ * Copy user data from/to a page which is mapped into a different
+ * processes address space.  Really, we want to allow our "user
+ * space" model to handle this.
+ */
+extern void copy_to_user_page(struct vm_area_struct *, struct page *,
+	unsigned long, void *, const void *, unsigned long);
+#define copy_from_user_page(vma, page, vaddr, dst, src, len) \
+	do {							\
+		memcpy(dst, src, len);				\
+	} while (0)
+
+#define flush_cache_dup_mm(mm) flush_cache_mm(mm)
+
+/*
+ * flush_dcache_page is used when the kernel has written to the page
+ * cache page at virtual address page->virtual.
+ *
+ * If this page isn't mapped (ie, page_mapping == NULL), or it might
+ * have userspace mappings, then we _must_ always clean + invalidate
+ * the dcache entries associated with the kernel mapping.
+ *
+ * Otherwise we can defer the operation, and clean the cache when we are
+ * about to change to user space.  This is the same method as used on SPARC64.
+ * See update_mmu_cache for the user space part.
+ */
+#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1
+extern void flush_dcache_page(struct page *);
+
+static inline void __flush_icache_all(void)
+{
+	asm("ic	ialluis");

arch/arm64/include/asm/cacheflush.h new file +37 -0

+}
+
+#define flush_dcache_mmap_lock(mapping) \
+	spin_lock_irq(&(mapping)->tree_lock)
+#define flush_dcache_mmap_unlock(mapping) \
+	spin_unlock_irq(&(mapping)->tree_lock)
+
+#define flush_icache_user_range(vma,page,addr,len) \
+	flush_dcache_page(page)
+
+/*
+ * We don't appear to need to do anything here.  In fact, if we did, we'd
+ * duplicate cache flushing elsewhere performed by flush_dcache_page().
+ */
+#define flush_icache_page(vma,page)	do { } while (0)
+
+/*
+ * flush_cache_vmap() is used when creating mappings (eg, via vmap,
+ * vmalloc, ioremap etc) in kernel space for pages.  On non-VIPT
+ * caches, since the direct-mappings of these pages may contain cached
+ * data, we need to do a full cache flush to ensure that writebacks
+ * don't corrupt data placed into these pages via the new mappings.
+ */
+static inline void flush_cache_vmap(unsigned long start, unsigned long end)
+{
+	/*
+	 * set_pte_at() called from vmap_pte_range() does not
+	 * have a DSB after cleaning the cache line.
+	 */
+	dsb();
+}
+
+static inline void flush_cache_vunmap(unsigned long start, unsigned long end)
+{
+}
+
+#endif

arch/arm64/include/asm/cachetype.h new file +48 -0

new file mode 100644
index 0000000..85f5f51
@@ -0,0 +1,48 @@
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ASM_CACHETYPE_H
+#define __ASM_CACHETYPE_H
+
+#include <asm/cputype.h>
+
+#define CTR_L1IP_SHIFT		14
+#define CTR_L1IP_MASK		3
+
+#define ICACHE_POLICY_RESERVED	0
+#define ICACHE_POLICY_AIVIVT	1
+#define ICACHE_POLICY_VIPT	2
+#define ICACHE_POLICY_PIPT	3
+
+static inline u32 icache_policy(void)
+{
+	return (read_cpuid_cachetype() >> CTR_L1IP_SHIFT) & CTR_L1IP_MASK;
+}
+
+/*
+ * Whilst the D-side always behaves as PIPT on AArch64, aliasing is
+ * permitted in the I-cache.
+ */
+static inline int icache_is_aliasing(void)
+{
+	return icache_policy() != ICACHE_POLICY_PIPT;
+}
+
+static inline int icache_is_aivivt(void)
+{
+	return icache_policy() == ICACHE_POLICY_AIVIVT;
+}
+
+#endif	/* __ASM_CACHETYPE_H */

arch/arm64/mm/cache.S new file +139 -0

new file mode 100644
index 0000000..3df0aa7
@@ -0,0 +1,251 @@
+/*
+ * Cache maintenance
+ *
+ * Copyright (C) 2001 Deep Blue Solutions Ltd.
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/linkage.h>
+#include <linux/init.h>
+#include <asm/assembler.h>
+
+#include "proc-macros.S"
+
+/*
+ *	__flush_dcache_all()
+ *
+ *	Flush the whole D-cache.
+ *
+ *	Corrupted registers: x0-x7, x9-x11
+ */
+ENTRY(__flush_dcache_all)
+	dsb	sy				// ensure ordering with previous memory accesses
+	mrs	x0, clidr_el1			// read clidr
+	and	x3, x0, #0x7000000		// extract loc from clidr
+	lsr	x3, x3, #23			// left align loc bit field
+	cbz	x3, finished			// if loc is 0, then no need to clean
+	mov	x10, #0				// start clean at cache level 0
+loop1:
+	add	x2, x10, x10, lsr #1		// work out 3x current cache level
+	lsr	x1, x0, x2			// extract cache type bits from clidr
+	and	x1, x1, #7			// mask of the bits for current cache only
+	cmp	x1, #2				// see what cache we have at this level
+	b.lt	skip				// skip if no cache, or just i-cache
+	save_and_disable_irqs x9		// make CSSELR and CCSIDR access atomic
+	msr	csselr_el1, x10			// select current cache level in csselr
+	isb					// isb to sych the new cssr&csidr
+	mrs	x1, ccsidr_el1			// read the new ccsidr
+	restore_irqs x9
+	and	x2, x1, #7			// extract the length of the cache lines
+	add	x2, x2, #4			// add 4 (line length offset)
+	mov	x4, #0x3ff
+	and	x4, x4, x1, lsr #3		// find maximum number on the way size
+	clz	x5, x4				// find bit position of way size increment
+	mov	x7, #0x7fff
+	and	x7, x7, x1, lsr #13		// extract max number of the index size
+loop2:
+	mov	x9, x4				// create working copy of max way size
+loop3:
+	lsl	x6, x9, x5
+	orr	x11, x10, x6			// factor way and cache number into x11
+	lsl	x6, x7, x2
+	orr	x11, x11, x6			// factor index number into x11
+	dc	cisw, x11			// clean & invalidate by set/way
+	subs	x9, x9, #1			// decrement the way
+	b.ge	loop3
+	subs	x7, x7, #1			// decrement the index
+	b.ge	loop2
+skip:
+	add	x10, x10, #2			// increment cache number
+	cmp	x3, x10
+	b.gt	loop1
+finished:
+	mov	x10, #0				// swith back to cache level 0
+	msr	csselr_el1, x10			// select current cache level in csselr
+	dsb	sy
+	isb
+	ret
+ENDPROC(__flush_dcache_all)
+
+/*
+ *	flush_cache_all()
+ *
+ *	Flush the entire cache system.  The data cache flush is now achieved
+ *	using atomic clean / invalidates working outwards from L1 cache. This
+ *	is done using Set/Way based cache maintainance instructions.  The
+ *	instruction cache can still be invalidated back to the point of
+ *	unification in a single instruction.
+ */
+ENTRY(flush_cache_all)
+	mov	x12, lr
+	bl	__flush_dcache_all
+	mov	x0, #0
+	ic	ialluis				// I+BTB cache invalidate
+	ret	x12
+ENDPROC(flush_cache_all)
+
+/*
+ *	flush_icache_range(start,end)
+ *
+ *	Ensure that the I and D caches are coherent within specified region.
+ *	This is typically used when code has been written to a memory region,
+ *	and will be executed.
+ *
+ *	- start   - virtual start address of region
+ *	- end     - virtual end address of region
+ */
+ENTRY(flush_icache_range)
+	/* FALLTHROUGH */
+
+/*
+ *	__flush_cache_user_range(start,end)
+ *
+ *	Ensure that the I and D caches are coherent within specified region.
+ *	This is typically used when code has been written to a memory region,
+ *	and will be executed.
+ *
+ *	- start   - virtual start address of region
+ *	- end     - virtual end address of region
+ */
+ENTRY(__flush_cache_user_range)
+	dcache_line_size x2, x3
+	sub	x3, x2, #1
+	bic	x4, x0, x3
+1:
+USER(9f, dc	cvau, x4	)		// clean D line to PoU
+	add	x4, x4, x2
+	cmp	x4, x1
+	b.lo	1b
+	dsb	sy
+
+	icache_line_size x2, x3
+	sub	x3, x2, #1
+	bic	x4, x0, x3
+1:
+USER(9f, ic	ivau, x4	)		// invalidate I line PoU
+	add	x4, x4, x2
+	cmp	x4, x1

arch/arm64/mm/cache.S new file +99 -0

+ *
+ *	- kaddr   - kernel address
+ *	- size    - size in question
+ */
+ENTRY(__flush_dcache_area)
+	dcache_line_size x2, x3
+	add	x1, x0, x1
+	sub	x3, x2, #1
+	bic	x0, x0, x3
+1:	dc	civac, x0			// clean & invalidate D line / unified line
+	add	x0, x0, x2
+	cmp	x0, x1
+	b.lo	1b
+	dsb	sy
+	ret
+ENDPROC(__flush_dcache_area)
+
+/*
+ *	dmac_inv_range(start,end)
+ *
+ *	Invalidate the data cache within the specified region; we will be
+ *	performing a DMA operation in this region and we want to purge old
+ *	data in the cache.
+ *
+ *	- start   - virtual start address of region
+ *	- end     - virtual end address of region
+ */
+ENTRY(dmac_inv_range)
+	dcache_line_size x2, x3
+	sub	x3, x2, #1
+	bic	x0, x0, x3
+	bic	x1, x1, x3
+1:	dc	ivac, x0			// invalidate D / U line
+	add	x0, x0, x2
+	cmp	x0, x1
+	b.lo	1b
+	dsb	sy
+	ret
+ENDPROC(dmac_inv_range)
+
+/*
+ *	dmac_clean_range(start,end)
+ *	- start   - virtual start address of region
+ *	- end     - virtual end address of region
+ */
+ENTRY(dmac_clean_range)
+	dcache_line_size x2, x3
+	sub	x3, x2, #1
+	bic	x0, x0, x3
+1:	dc	cvac, x0			// clean D / U line
+	add	x0, x0, x2
+	cmp	x0, x1
+	b.lo	1b
+	dsb	sy
+	ret
+ENDPROC(dmac_clean_range)
+
+/*
+ *	dmac_flush_range(start,end)
+ *	- start   - virtual start address of region
+ *	- end     - virtual end address of region
+ */
+ENTRY(dmac_flush_range)
+	dcache_line_size x2, x3
+	sub	x3, x2, #1
+	bic	x0, x0, x3
+1:	dc	civac, x0			// clean & invalidate D / U line
+	add	x0, x0, x2
+	cmp	x0, x1
+	b.lo	1b
+	dsb	sy
+	ret
+ENDPROC(dmac_flush_range)
+
+/*
+ *	dmac_map_area(start, size, dir)
+ *	- start	- kernel virtual start address
+ *	- size	- size of region
+ *	- dir	- DMA direction
+ */
+ENTRY(dmac_map_area)
+	add	x1, x1, x0
+	cmp	x2, #DMA_FROM_DEVICE
+	b.eq	dmac_inv_range
+	b	dmac_clean_range
+ENDPROC(dmac_map_area)
+
+/*
+ *	dmac_unmap_area(start, size, dir)
+ *	- start	- kernel virtual start address
+ *	- size	- size of region
+ *	- dir	- DMA direction
+ */
+ENTRY(dmac_unmap_area)
+	add	x1, x1, x0
+	cmp	x2, #DMA_TO_DEVICE
+	b.ne	dmac_inv_range
+	ret
+ENDPROC(dmac_unmap_area)

arch/arm64/mm/flush.c new file +114 -0

new file mode 100644
index 0000000..6138d86
@@ -0,0 +1,138 @@
+/*
+ * Based on arch/arm/mm/flush.c
+ *
+ * Copyright (C) 1995-2002 Russell King
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/export.h>
+#include <linux/mm.h>
+#include <linux/pagemap.h>
+
+#include <asm/cacheflush.h>
+#include <asm/cachetype.h>
+#include <asm/tlbflush.h>
+
+#include "mm.h"
+
+void flush_cache_mm(struct mm_struct *mm)
+{
+}
+
+void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
+		       unsigned long end)
+{
+	if (vma->vm_flags & VM_EXEC)
+		__flush_icache_all();
+}
+
+void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr,
+		      unsigned long pfn)
+{
+}
+
+static void flush_ptrace_access(struct vm_area_struct *vma, struct page *page,
+				unsigned long uaddr, void *kaddr,
+				unsigned long len)
+{
+	if (vma->vm_flags & VM_EXEC) {
+		unsigned long addr = (unsigned long)kaddr;
+		if (icache_is_aliasing()) {
+			__flush_dcache_area(kaddr, len);
+			__flush_icache_all();
+		} else {
+			flush_icache_range(addr, addr + len);
+		}
+	}
+}
+
+/*
+ * Copy user data from/to a page which is mapped into a different processes
+ * address space.  Really, we want to allow our "user space" model to handle
+ * this.
+ *
+ * Note that this code needs to run on the current CPU.
+ */
+void copy_to_user_page(struct vm_area_struct *vma, struct page *page,
+		       unsigned long uaddr, void *dst, const void *src,
+		       unsigned long len)
+{
+#ifdef CONFIG_SMP
+	preempt_disable();
+#endif
+	memcpy(dst, src, len);
+	flush_ptrace_access(vma, page, uaddr, dst, len);
+#ifdef CONFIG_SMP
+	preempt_enable();
+#endif
+}
+
+void __flush_dcache_page(struct address_space *mapping, struct page *page)
+{
+	__flush_dcache_area(page_address(page), PAGE_SIZE);
+}
+
+void __sync_icache_dcache(pte_t pte)
+{
+	unsigned long pfn;
+	struct page *page;
+
+	pfn = pte_pfn(pte);
+	if (!pfn_valid(pfn))
+		return;
+
+	page = pfn_to_page(pfn);
+	if (!test_and_set_bit(PG_dcache_clean, &page->flags))
+		__flush_dcache_page(NULL, page);
+	__flush_icache_all();
+}
+
+/*
+ * Ensure cache coherency between kernel mapping and userspace mapping of this
+ * page.
+ */
+void flush_dcache_page(struct page *page)
+{
+	struct address_space *mapping;
+
+	/*
+	 * The zero page is never written to, so never has any dirty cache
+	 * lines, and therefore never needs to be flushed.
+	 */
+	if (page == ZERO_PAGE(0))

arch/arm64/include/asm/cacheflush.h line 131

Quoted 8 lines · Catalin Marinas

arch/arm64/include/asm/cacheflush.h new file +8 -0

+#define ARCH_HAS_FLUSH_ANON_PAGE
+static inline void flush_anon_page(struct vm_area_struct *vma,
+                      struct page *page, unsigned long vmaddr)
+{
+     extern void __flush_anon_page(struct vm_area_struct *vma,
+                             struct page *, unsigned long);
+     if (PageAnon(page))
+             __flush_anon_page(vma, page, vmaddr);

__flush_anon_page() does nothing. Shouldn't this be removed as well?

Quoted 33 lines

arch/arm64/mm/flush.c new file +33 -0

+void __flush_dcache_page(struct address_space *mapping, struct page *page)
+{
+     __flush_dcache_area(page_address(page), PAGE_SIZE);
+}
+
+void __sync_icache_dcache(pte_t pte)
+{
+     unsigned long pfn;
+     struct page *page;
+
+     pfn = pte_pfn(pte);
+     if (!pfn_valid(pfn))
+             return;
+
+     page = pfn_to_page(pfn);
+     if (!test_and_set_bit(PG_dcache_clean, &page->flags))
+             __flush_dcache_page(NULL, page);
+     __flush_icache_all();
+}
+
+/*
+ * Ensure cache coherency between kernel mapping and userspace mapping of this
+ * page.
+ */
+void flush_dcache_page(struct page *page)
+{
+     struct address_space *mapping;
+
+     /*
+      * The zero page is never written to, so never has any dirty cache
+      * lines, and therefore never needs to be flushed.
+      */
+     if (page == ZERO_PAGE(0))

arch/arm64/mm/flush.c line 124

Quoted 10 lines

arch/arm64/mm/flush.c new file +10 -0

+             return;
+
+     mapping = page_mapping(page);
+
+     if (mapping && !mapping_mapped(mapping))
+             clear_bit(PG_dcache_clean, &page->flags);
+     else {
+             __flush_dcache_page(mapping, page);
+             if (mapping)
+                     __flush_icache_all();

Is this necessary to ensure I/D coherency? Then, I would have expected

                if (mapping) {
                        __flush_dcache_page(mapping, page);
                        __flush_icache_all();
                }

similar to __sync_icache_dcache() above.

What is the reason why the D-cache flush is done in different cases than the following I-cache flush?

Quoted 6 lines

arch/arm64/mm/flush.c new file +6 -0

+             set_bit(PG_dcache_clean, &page->flags);
+     }
+}
+EXPORT_SYMBOL(flush_dcache_page);
+
+void __flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned long vmaddr)

Review

Quoted 2 lines

(quoted diff) +2 -0

+{
+}

Note that the __flush_dcache_page(mapping, page) in flush_dcache_page() above is called when page is an anonymous page (since mapping == NULL in this case). If the call to __flush_dcache_page() is right above, it should be needed here as well?

Quoted 49 lines

Quoted 34 lines

arch/arm64/mm/flush.c new file +33 -0

+void __flush_dcache_page(struct address_space *mapping, struct page *page)
+{
+     __flush_dcache_area(page_address(page), PAGE_SIZE);
+}
+
+void __sync_icache_dcache(pte_t pte)
+{
+     unsigned long pfn;
+     struct page *page;
+
+     pfn = pte_pfn(pte);
+     if (!pfn_valid(pfn))
+             return;
+
+     page = pfn_to_page(pfn);
+     if (!test_and_set_bit(PG_dcache_clean, &page->flags))
+             __flush_dcache_page(NULL, page);
+     __flush_icache_all();
+}
+
+/*
+ * Ensure cache coherency between kernel mapping and userspace mapping of this
+ * page.
+ */
+void flush_dcache_page(struct page *page)
+{
+     struct address_space *mapping;
+
+     /*
+      * The zero page is never written to, so never has any dirty cache
+      * lines, and therefore never needs to be flushed.
+      */
+     if (page == ZERO_PAGE(0))

arch/arm64/mm/flush.c line 124

Quoted 10 lines

arch/arm64/mm/flush.c new file +10 -0

+             return;
+
+     mapping = page_mapping(page);
+
+     if (mapping && !mapping_mapped(mapping))
+             clear_bit(PG_dcache_clean, &page->flags);
+     else {
+             __flush_dcache_page(mapping, page);
+             if (mapping)
+                     __flush_icache_all();

Is this necessary to ensure I/D coherency? Then, I would have expected

Quoted 6 lines

                if (mapping) {
                        __flush_dcache_page(mapping, page);
                        __flush_icache_all();
                }

similar to __sync_icache_dcache() above.

We don't want to do additional flushing if !mapping_mapped() as the page isn't mapped in user space. In this case we defer the flushing until __sync_icache_dcache().

The other case is for anonymous pages where mapping == NULL. Here we don't defer the D-cache flush and do it directly. The I-cache, if needed, is handled later in __sync_icache_dcache(). This was based on the idea that this case is mainly for the args/env page which is mapped shortly after anyway, so not worth deferring. On AArch64, I don't think it makes any difference. Maybe a slight improvement (at least in clarity) in flush_dcache_page():

	if (mapping && mapping_mapped(mapping)) {
		__flush_dcache_page(page);
		__flush_icache_all();
		set_bit(PG_dcache_clean, &page->flags);
	} else {
		clear_bit(PG_dcache_clean, &page->flags);
	}

In this case the anonymous page flushing is deferred to __sync_icache_dcache().

Quoted 2 lines

What is the reason why the D-cache flush is done in different cases than the following I-cache flush?

For __sync_icache_dcache(), we need to handle the situation where the page mapped into user space has been cleaned (D-cache) but there may be stale data in the I-cache. I think this can only happen with an ASID-tagged VIVT I-cache configuration (which is allowed on AArch64) if an existing page has been unmapped and the same virtual address remapped (withing the same mm context) to a different page that had been cleaned previously. We could optimise the __sync_icache_dcache() as below:

	if (!test_and_set_bit(PG_dcache_clean, &page->flags)) {
		__flush_dcache_page(page);
		__flush_icache_all();
	} else if (icache_is_aivivt()) {
		__flush_icache_all();
	}

arch/arm64/mm/cache.S new file +0 -0

new file mode 100644
index 0000000..3df0aa7

+/*
+ * __flush_kern_dcache_page(kaddr)

+ *
+ * Ensure that the data held in the page kaddr is written back
to the
+ * page in question.

+ *
+ * - kaddr - kernel address
+ * - size - size in question
+ */

arch/arm64/mm/cache.S new file +4 -0

+ENTRY(__flush_dcache_area)
+	dcache_line_size x2, x3
+	add	x1, x0, x1
+

Quoted 75 lines · Catalin Marinas

Quoted 48 lines

Quoted 33 lines

arch/arm64/mm/flush.c new file +33 -0

+void __flush_dcache_page(struct address_space *mapping, struct page *page)
+{
+     __flush_dcache_area(page_address(page), PAGE_SIZE);
+}
+
+void __sync_icache_dcache(pte_t pte)
+{
+     unsigned long pfn;
+     struct page *page;
+
+     pfn = pte_pfn(pte);
+     if (!pfn_valid(pfn))
+             return;
+
+     page = pfn_to_page(pfn);
+     if (!test_and_set_bit(PG_dcache_clean, &page->flags))
+             __flush_dcache_page(NULL, page);
+     __flush_icache_all();
+}
+
+/*
+ * Ensure cache coherency between kernel mapping and userspace mapping of this
+ * page.
+ */
+void flush_dcache_page(struct page *page)
+{
+     struct address_space *mapping;
+
+     /*
+      * The zero page is never written to, so never has any dirty cache
+      * lines, and therefore never needs to be flushed.
+      */
+     if (page == ZERO_PAGE(0))

arch/arm64/mm/flush.c line 124

Quoted 10 lines

arch/arm64/mm/flush.c new file +10 -0

+             return;
+
+     mapping = page_mapping(page);
+
+     if (mapping && !mapping_mapped(mapping))
+             clear_bit(PG_dcache_clean, &page->flags);
+     else {
+             __flush_dcache_page(mapping, page);
+             if (mapping)
+                     __flush_icache_all();

Is this necessary to ensure I/D coherency? Then, I would have expected

Quoted 6 lines

                if (mapping) {
                        __flush_dcache_page(mapping, page);
                        __flush_icache_all();
                }

similar to __sync_icache_dcache() above.

We don't want to do additional flushing if !mapping_mapped() as the page isn't mapped in user space. In this case we defer the flushing until __sync_icache_dcache().

The other case is for anonymous pages where mapping == NULL. Here we don't defer the D-cache flush and do it directly. The I-cache, if needed, is handled later in __sync_icache_dcache(). This was based on the idea that this case is mainly for the args/env page which is mapped shortly after anyway, so not worth deferring. On AArch64, I don't think it makes any difference. Maybe a slight improvement (at least in clarity) in flush_dcache_page():

	if (mapping && mapping_mapped(mapping)) {
		__flush_dcache_page(page);
		__flush_icache_all();
		set_bit(PG_dcache_clean, &page->flags);
	} else {
		clear_bit(PG_dcache_clean, &page->flags);
	}

Quoted 2 lines

In this case the anonymous page flushing is deferred to __sync_icache_dcache().

Yes, I think this is much clearer. It makes clear that the D-cache flush is done to avoid I/D incoherencies. Previously, the obvious question was "Why do we flush only the D-cache in some situations given that it essentially behaves like a PIPT cache?" (the motivation for this became only clear to me after your explanation.)

However, the reason why this was an obvious question for me is
interesting: I think the main use case you had in mind does not happen since about five years (since commit b6a2fe, "mm: variable length argument support"). And I had a completely different main use case in mind.

I reckon that the use case you refer to is the use in fs/exec.c? Copying arg/env was changed in the commit mentioned above. This also changed the use of flush_dcache_page() (which is not supposed to handle anon pages) into flush_kernel_dcache_page() (which is supposed to handle kernel modified user pages, i.e. also anon pages).

In case of direct I/O (and probably also in other cases like SG_IO) the block layer will see pages from get_user_pages() directly, i.e. also anonymous pages. Many drivers (especially emulated storage drivers like dm-crypt) use flush_dcache_page() after modifying a page. Although flush_dcache_page() is not even supposed to handle anonymous pages, it flushes the kernel mapping of the page because of this code line and everything is well on aliasing D-caches.

Back to arm64 (and possibly to arm with non-aliasing D-caches?), this also means that the saved D-cache flush in the anonymous page case is not only a slight improvement on clarity, but may avoid a considerable number of D-cache flushes in some I/O situations. (If it is still correct that there are no problems with the I-cache for this use case.)

If now we could additionally avoid to flush the entire I-cache for every page in direct I/O operations with user mapped page cache pages (e.g. direct I/O read into an mmap region)...

Quoted 11 lines

Quoted 2 lines

What is the reason why the D-cache flush is done in different cases than the following I-cache flush?

For __sync_icache_dcache(), we need to handle the situation where the page mapped into user space has been cleaned (D-cache) but there may be stale data in the I-cache. I think this can only happen with an ASID-tagged VIVT I-cache configuration (which is allowed on AArch64) if an existing page has been unmapped and the same virtual address remapped (withing the same mm context) to a different page that had been cleaned previously. We could optimise the __sync_icache_dcache() as below:

Quoted 6 lines

	if (!test_and_set_bit(PG_dcache_clean, &page->flags)) {
		__flush_dcache_page(page);
		__flush_icache_all();
	} else if (icache_is_aivivt()) {
		__flush_icache_all();
	}

Sorry, this is out of my depth. I think I don't really understand the cases leading to I/D incoherency.

Quoted 1 line · Simon Baatz

On Wed, Sep 12, 2012 at 10:29:54AM +0100, Catalin Marinas wrote:

...

Quoted 7 lines

In case of direct I/O (and probably also in other cases like SG_IO) the block layer will see pages from get_user_pages() directly, i.e. also anonymous pages. Many drivers (especially emulated storage drivers like dm-crypt) use flush_dcache_page() after modifying a page. Although flush_dcache_page() is not even supposed to handle anonymous pages, it flushes the kernel mapping of the page because of this code line and everything is well on aliasing D-caches.

According to the cachetlb.txt document (though not sure architecture ports follow it entirely), flush_dcache_page() deliberately shouldn't follow anonymous pages. But it seems that we do it on ARM (maybe as an alternative to flush_kernel_dcache_page()).

Quoted 6 lines

Back to arm64 (and possibly to arm with non-aliasing D-caches?), this also means that the saved D-cache flush in the anonymous page case is not only a slight improvement on clarity, but may avoid a considerable number of D-cache flushes in some I/O situations. (If it is still correct that there are no problems with the I-cache for this use case.)

The I-cache would be needed if the kernel modifies an executable user page. But I don't see a case for this yet. So with non-aliasing D-cache the flush_kernel_dcache_page() can be a nop.

Quoted 3 lines

If now we could additionally avoid to flush the entire I-cache for every page in direct I/O operations with user mapped page cache pages (e.g. direct I/O read into an mmap region)...

If the page is already mapped, we don't have a later hook to be able to flush the caches, so we do it here. We can avoid the I-cache operation only if we are sure that the user would not execute code from such page. IOW the direct I/O wouldn't write any instructions.

The powerpc implementation of flush_dcache_page() doesn't even check for the existence of a mapping, it always marks the page as dirty. We can do the same on arm64 (only leave the clear_bit part of the condition) as long as we know that the kernel wouldn't write new code into a page that is already mapped.