Skip to contentSkip to conversation

iommu/vt-d: Remove unnecassary qi clflushes

PATCH Applied Nadav Amit 4 participants 8 messages Jun 24, 2016 – Jul 13, 2016 6 messages not in the archive IOMMU

lore
Share & tools
lore

Nadav Amit

author
#0 lore raw

PATCH 1/1 iommu/vt-d: Remove unnecassary qi clflushes

According to the manual: "Hardware access to ... invalidation queue ... are always coherent."

Remove unnecassary clflushes accordingly.

---

Build-tested since I do not have an IOMMU that does not support coherency.

---
 drivers/iommu/dmar.c | 5 -----
 1 file changed, 5 deletions(-)
drivers/iommu/dmar.c +0 -5
index 6a86b5d..ca56ec3 100644
@@ -1155,8 +1155,6 @@ static int qi_check_fault(struct intel_iommu *iommu, int index)
 				(unsigned long long)qi->desc[index].high);
 			memcpy(&qi->desc[index], &qi->desc[wait_index],
 					sizeof(struct qi_desc));
-			__iommu_flush_cache(iommu, &qi->desc[index],
-					sizeof(struct qi_desc));
 			writel(DMA_FSTS_IQE, iommu->reg + DMAR_FSTS_REG);
 			return -EINVAL;
 		}
@@ -1231,9 +1229,6 @@ restart:
 
 	hw[wait_index] = wait_desc;
 
-	__iommu_flush_cache(iommu, &hw[index], sizeof(struct qi_desc));
-	__iommu_flush_cache(iommu, &hw[wait_index], sizeof(struct qi_desc));
-
 	qi->free_head = (qi->free_head + 2) % QI_LENGTH;
 	qi->free_cnt -= 2;
 
Signature · 1 line
2.7.4

A message that is not in the archive (sent off-list or not yet received): 1466773994-242872-1-git-send-email-namit-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org

#2 lore raw
Quoted 4 lines · Nadav Amit

According to the manual: "Hardware access to ... invalidation queue ... are always coherent."

Remove unnecassary clflushes accordingly.

It is one thing what the spec says and another how hardware really behaves. Have you tested this on (potentially really old) VT-d machines to make sure the spec is _always_ right here?

Joerg

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 9 lines · Joerg Roedel
Quoted 4 lines · Nadav Amit

According to the manual: "Hardware access to ... invalidation queue ... are always coherent."

Remove unnecassary clflushes accordingly.

It is one thing what the spec says and another how hardware really behaves. Have you tested this on (potentially really old) VT-d machines to make sure the spec is _always_ right here?

No I didn’t as the commit message says. I would be happy for someones’ tested-by.

Having said that - FreeBSD does not do these (unnecessary) invalidations [1], and their code comment clearly says it. Since this code is not new, I would assume FreeBSD would crash by now if the code was buggy.

Although such hardware is old, there are some hypervisors that do not set the ecap.coherency of emulated IOMMUs. Yes, it is unwise, but there is no reason to further punish these hypervisors.

Thanks,
Nadav

[1] http://web.mit.edu/freebsd/head/sys/x86/iommu/intel_qi.c
_______________________________________________
iommu mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

A message that is not in the archive (sent off-list or not yet received): C78095BC-3BED-48DE-90CE-601DE40AA613-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#6 lore raw
Quoted 15 lines · Nadav Amit
Quoted 9 lines · Joerg Roedel
Quoted 4 lines · Nadav Amit

According to the manual: "Hardware access to ... invalidation queue ... are always coherent."

Remove unnecassary clflushes accordingly.

It is one thing what the spec says and another how hardware really behaves. Have you tested this on (potentially really old) VT-d machines to make sure the spec is _always_ right here?

No I didn’t as the commit message says. I would be happy for someones’ tested-by.

Quoted 8 lines

Having said that - FreeBSD does not do these (unnecessary) invalidations [1], and their code comment clearly says it. Since this code is not new, I would assume FreeBSD would crash by now if the code was buggy.

Although such hardware is old, there are some hypervisors that do not set the ecap.coherency of emulated IOMMUs. Yes, it is unwise, but there is no reason to further punish these hypervisors.

Ping? To clarify - this behavior applies to the guest of VMware and KVM (which uses QEMU for IOMMU emulation).

Regardless, please respond to the other patch I sent as well.

Thanks,
Nadav

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

A message that is not in the archive (sent off-list or not yet received): 926A8CFB-D22D-4DC6-9EE0-D912451365BB-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#8 lore raw
Quoted 3 lines · Nadav Amit
Quoted 3 lines

Although such hardware is old, there are some hypervisors that do not set the ecap.coherency of emulated IOMMUs. Yes, it is unwise, but there is no reason to further punish these hypervisors.

QEMU will need the kernel to respect ecap.coherency in order to support nested VFIO, for example.

Paolo

A message that is not in the archive (sent off-list or not yet received): ac3ff8d4-e47b-e4d8-40ed-02811732526a-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org

#10 lore raw
Quoted 9 lines · Paolo Bonzini
Quoted 3 lines · Nadav Amit
Quoted 3 lines

Although such hardware is old, there are some hypervisors that do not set the ecap.coherency of emulated IOMMUs. Yes, it is unwise, but there is no reason to further punish these hypervisors.

QEMU will need the kernel to respect ecap.coherency in order to support nested VFIO, for example.

To clarify - the kernel respects the coherency, but performs more clflushes than necessary. It has no functional impact, but induces performance degradation (which I did not measure, but is likely to be several hundreds of cycles per flush).

Regards,
Nadav

A message that is not in the archive (sent off-list or not yet received): 5F48A3E6-99EE-4DC5-BF38-4F0338E9E8D0-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#12 lore raw
Quoted 9 lines · Nadav Amit
Quoted 7 lines · Paolo Bonzini
Quoted 3 lines · Nadav Amit
Quoted 3 lines

Although such hardware is old, there are some hypervisors that do not set the ecap.coherency of emulated IOMMUs. Yes, it is unwise, but there is no reason to further punish these hypervisors.

Quoted 7 lines
Quoted 2 lines

QEMU will need the kernel to respect ecap.coherency in order to support nested VFIO, for example.

To clarify - the kernel respects the coherency, but performs more clflushes than necessary. It has no functional impact, but induces performance degradation (which I did not measure, but is likely to be several hundreds of cycles per flush).

Oh, then QEMU indeed doesn't care about flushing the invalidation queue.

Paolo