Skip to contentSkip to conversation

updated: arm64: KVM: vgic: deal with GIC sub-page alignment

PATCH In review Itaru Kitayama 2 participants 10 messages Jul 1, 2016 – Jul 10, 2016

Subsystems (2)
arm64 KVM arm64
lore
Share & tools
lore

Itaru Kitayama

author
#0 lore raw

PATCH 1/1 updated: arm64: KVM: vgic: deal with GIC sub-page alignment

This is merely an update to the patch Marc posted in 2014:

https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/010055.html

This updated patch is against today's current kvmarm git tree tag, kvm-for-arm-v4.7-rc6.

The GIC CPU interface is always 4k aligned. If the host is using 64k pages, it is critical to place the guest's GICC interface at the same relative alignment as the host's GICV. Failure to do so results in an impossibility for the guest to deal with interrupts.

Add a KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET attribute for the VGIC, allowing userspace to retrieve the GICV offset in a page. It becomes then trivial to adjust the GICC base address for the guest.

Signed-off-by Marc Zyngier <[email protected] <https://lists.cs.columbia.edu/mailman/listinfo/kvmarm>>
Signed-off-by Itaru Kitayama <[email protected]>
arch/arm64/include/uapi/asm/kvm.h +0 -0
index f209ea1..803b097 100644
@@ -203,6 +203,7 @@ struct kvm_arch_memory_slot {
 #define   KVM_DEV_ARM_VGIC_OFFSET_MASK (0xffffffffULL <<

KVM_DEV_ARM_VGIC_OFFSET_SHIFT)
#define KVM_DEV_ARM_VGIC_GRP_NR_IRQS 3
#define KVM_DEV_ARM_VGIC_GRP_CTRL 4
+#define KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET 5
#define KVM_DEV_ARM_VGIC_CTRL_INIT 0

/* Device Control API on vcpu fd */

virt/kvm/arm/vgic/vgic-kvm-device.c +7 -0
index 850eb88..3e33778 100644
@@ -196,8 +196,14 @@ static int vgic_get_common_attr(struct kvm_device *dev,
                             VGIC_NR_PRIVATE_IRQS, uaddr);
                break;
        }
+       case KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET: {
+               u32 __user *uaddr = (u32 __user *)(long)attr->addr;
+               u32 val = kvm_vgic_global_state.vcpu_base & ~PAGE_MASK;
+               r = put_user(val, uaddr);
+               break;
        }
 
+       }
        return r;
 }
 

@@ -362,6 +368,7 @@ static int vgic_v2_has_attr(struct kvm_device *dev,
        case KVM_DEV_ARM_VGIC_GRP_CPU_REGS:
                return vgic_v2_has_attr_regs(dev, attr);
        case KVM_DEV_ARM_VGIC_GRP_NR_IRQS:
+       case KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET:
                return 0;
        case KVM_DEV_ARM_VGIC_GRP_CTRL:
                switch (attr->attr) {

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#1 lore raw

Itaru,

Quoted 12 lines · Itaru Kitayama

This is merely an update to the patch Marc posted in 2014:

https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/010055.html

This updated patch is against today's current kvmarm git tree tag, kvm-for-arm-v4.7-rc6.

The GIC CPU interface is always 4k aligned. If the host is using 64k pages, it is critical to place the guest's GICC interface at the same relative alignment as the host's GICV. Failure to do so results in an impossibility for the guest to deal with interrupts.

Quoted 6 lines

Add a KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET attribute for the VGIC, allowing userspace to retrieve the GICV offset in a page. It becomes then trivial to adjust the GICC base address for the guest.

Signed-off-by Marc Zyngier <[email protected] <https://lists.cs.columbia.edu/mailman/listinfo/kvmarm>>
Signed-off-by Itaru Kitayama <[email protected]>

I'm working on a much more complete patch series that deals with this and a bit more. Hopefully for the 4.9 time frame.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
author
#2 lore raw

Marc,
That's good news. Can I assume you'd keep the KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET attribute, or would you introduce new ones? I ask because either way we need the userland support, be it QEMU or kvmtool. I'm carrying the small QEMU patch at this moment.

Quoted 29 lines · Marc Zyngier

Itaru,

Quoted 12 lines · Itaru Kitayama

This is merely an update to the patch Marc posted in 2014:

https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/010055.html

This updated patch is against today's current kvmarm git tree tag, kvm-for-arm-v4.7-rc6.

The GIC CPU interface is always 4k aligned. If the host is using 64k pages, it is critical to place the guest's GICC interface at the same relative alignment as the host's GICV. Failure to do so results in an impossibility for the guest to deal with interrupts.

Quoted 6 lines

Add a KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET attribute for the VGIC, allowing userspace to retrieve the GICV offset in a page. It becomes then trivial to adjust the GICC base address for the guest.

Signed-off-by Marc Zyngier <[email protected] <https://lists.cs.columbia.edu/mailman/listinfo/kvmarm>>
Signed-off-by Itaru Kitayama <[email protected]>

I'm working on a much more complete patch series that deals with this and a bit more. Hopefully for the 4.9 time frame.

Thanks,

	M.
#3 lore raw
Quoted 5 lines · Itaru Kitayama

Marc,
That's good news. Can I assume you'd keep the KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET attribute, or would you introduce new ones? I ask because either way we need the userland support, be it QEMU or kvmtool. I'm carrying the small QEMU patch at this moment.

You can look at the kvm-arm64/gicv-align branch in my tree, which contains some of the stuff. At the moment, the attribute is named KVM_DEV_ARM_VGIC_GRP_GICC_OFFSET, but I don't mind changing the name.

Thanks,

	M.
Quoted 33 lines · Marc Zyngier
Quoted 29 lines

Itaru,

Quoted 12 lines · Itaru Kitayama

This is merely an update to the patch Marc posted in 2014:

https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/010055.html

This updated patch is against today's current kvmarm git tree tag, kvm-for-arm-v4.7-rc6.

The GIC CPU interface is always 4k aligned. If the host is using 64k pages, it is critical to place the guest's GICC interface at the same relative alignment as the host's GICV. Failure to do so results in an impossibility for the guest to deal with interrupts.

Quoted 6 lines

Add a KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET attribute for the VGIC, allowing userspace to retrieve the GICV offset in a page. It becomes then trivial to adjust the GICC base address for the guest.

Signed-off-by Marc Zyngier <[email protected] <https://lists.cs.columbia.edu/mailman/listinfo/kvmarm>>
Signed-off-by Itaru Kitayama <[email protected]>

I'm working on a much more complete patch series that deals with this and a bit more. Hopefully for the 4.9 time frame.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
author
#4 lore raw

Marc,

I'm at the latest commit, fa9301d in the gicv-align branch at the moment, with that an unmodified QEMU guest boots fine on a 64Kb page granular host, thanks to the trapping you introduced. (I wanted to get some performance statistics of the new feature with perf, but on arm64 most of them are unsupported)

However, if I rebuild the kernel using defconfig with only the kernel page size changed to 16Kb it does not boot at least on Overdrive 3000.

Quoted 14 lines · Marc Zyngier
Quoted 5 lines · Itaru Kitayama

Marc,
That's good news. Can I assume you'd keep the KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET attribute, or would you introduce new ones? I ask because either way we need the userland support, be it QEMU or kvmtool. I'm carrying the small QEMU patch at this moment.

You can look at the kvm-arm64/gicv-align branch in my tree, which contains some of the stuff. At the moment, the attribute is named KVM_DEV_ARM_VGIC_GRP_GICC_OFFSET, but I don't mind changing the name.

Thanks,

	M.
author
#5 lore raw

Forgot to mention that all tests/attempts were done on an ACPI-based system.

Quoted 27 lines · Itaru Kitayama

Marc,

I'm at the latest commit, fa9301d in the gicv-align branch at the moment, with that an unmodified QEMU guest boots fine on a 64Kb page granular host, thanks to the trapping you introduced.
(I wanted to get some performance statistics of the new feature with perf, but on arm64 most of them are unsupported)

However, if I rebuild the kernel using defconfig with only the kernel page size changed to 16Kb it does not boot at least on Overdrive 3000.

Quoted 14 lines · Marc Zyngier
Quoted 5 lines · Itaru Kitayama

Marc,
That's good news. Can I assume you'd keep the KVM_DEV_ARM_VGIC_GRP_ADDR_OFFSET attribute, or would you introduce new ones? I ask because either way we need the userland support, be it QEMU or kvmtool. I'm carrying the small QEMU patch at this moment.

You can look at the kvm-arm64/gicv-align branch in my tree, which contains some of the stuff. At the moment, the attribute is named KVM_DEV_ARM_VGIC_GRP_GICC_OFFSET, but I don't mind changing the name.

Thanks,

    M.
#6 lore raw
In reply to Itaru Kitayama

On Sat, 9 Jul 2016 09:27:21 +0900
Itaru Kitayama <[email protected]> wrote:

Itaru,

Quoted 3 lines

I'm at the latest commit, fa9301d in the gicv-align branch at the moment, with that an unmodified QEMU guest boots fine on a 64Kb page granular host, thanks to the trapping you introduced.

Please realize that this is a work in progress, and that I'm not even sure I want to upstream this.

Quoted 2 lines

(I wanted to get some performance statistics of the new feature with perf, but on arm64 most of them are unsupported)

Care to mention which ones are missing?

Quoted 2 lines

However, if I rebuild the kernel using defconfig with only the kernel page size changed to 16Kb it does not boot at least on Overdrive 3000.

Only a few CPUs support 16kB pages, and Cortex-A57 is definitely not one of them.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny.
author
#7 lore raw
Quoted 2 lines · Marc Zyngier

Please realize that this is a work in progress, and that I'm not even sure I want to upstream this.

OK.

Quoted 4 lines
Quoted 2 lines

(I wanted to get some performance statistics of the new feature with perf, but on arm64 most of them are unsupported)

Care to mention which ones are missing?

$ perf stat sleep 5

Performance counter stats for 'sleep 5':

          0.589480      task-clock:u (msec)       #    0.000 CPUs

utilized
0 context-switches:u # 0.000 K/sec

0 cpu-migrations:u # 0.000 K/sec

35 page-faults:u # 0.059 M/sec

<not supported> cycles:u

<not supported> instructions:u

<not supported> branches:u

<not supported> branch-misses:u

       5.024683421 seconds time elapsed

Userland is Fedora 24.

Quoted 2 lines

Only a few CPUs support 16kB pages, and Cortex-A57 is definitely not one of them.

Ah, I did not know that. Thanks.

#8 lore raw
Quoted 27 lines · On Sat, 9 Jul 2016 19:32:35 +0900 Itaru Kitayama <[email protected]>…
Quoted 2 lines · Marc Zyngier

Please realize that this is a work in progress, and that I'm not even sure I want to upstream this.

OK.

Quoted 4 lines
Quoted 2 lines

(I wanted to get some performance statistics of the new feature with perf, but on arm64 most of them are unsupported)

Care to mention which ones are missing?

$ perf stat sleep 5

Performance counter stats for 'sleep 5':

           0.589480      task-clock:u (msec)       #    0.000 CPUs

utilized
0 context-switches:u # 0.000 K/sec

0 cpu-migrations:u # 0.000 K/sec

35 page-faults:u # 0.059 M/sec

<not supported> cycles:u

<not supported> instructions:u

Quoted 8 lines

<not supported> branches:u

<not supported> branch-misses:u

5.024683421 seconds time elapsed

Userland is Fedora 24.

I'm pretty sure your QEMU doesn't have PMU support. Can you show the PMU probe messages in the guest? And/or dump the guest DTB?

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny.
author
#9 lore raw

I was issuing the command on host which was booted up with acpi=force.

Booting up the same system with acpi=off, the probing message is there

[    2.106248] hw perfevents: enabled with armv8_pmuv3 PMU driver, 7 counters available

and the support is there too:

$ perf stat sleep 1

Performance counter stats for 'sleep 1':

          0.624936      task-clock:u (msec)       #    0.001 CPUs

utilized
0 context-switches:u # 0.000 K/sec

0 cpu-migrations:u # 0.000 K/sec

35 page-faults:u # 0.056 M/sec

389,265 cycles:u # 0.623 GHz

184,079 instructions:u # 0.47 insn per
cycle
<not supported> branches:u

4,828 branch-misses:u # 0.00% of all
branches

       1.013477604 seconds time elapsed

I'll apply the PMU probing patches for ACPI platforms and see if that helps.

Quoted 33 lines · Marc Zyngier
Quoted 17 lines
$ perf stat sleep 5

Performance counter stats for 'sleep 5':

           0.589480      task-clock:u (msec)       #    0.000 CPUs

utilized
0 context-switches:u # 0.000 K/sec

0 cpu-migrations:u # 0.000 K/sec

35 page-faults:u # 0.059 M/sec

<not supported> cycles:u

<not supported> instructions:u

Quoted 8 lines

<not supported> branches:u

<not supported> branch-misses:u

5.024683421 seconds time elapsed

Userland is Fedora 24.

I'm pretty sure your QEMU doesn't have PMU support. Can you show the PMU probe messages in the guest? And/or dump the guest DTB?

Thanks,

	M.