Skip to contentSkip to conversation

vfio: Split the container code into a clean layer and dedicated file

PATCH Superseded Jason Gunthorpe 3 participants 30 messages Aug 31, 2022 – Sep 6, 2022

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover vfio: Split the container code into a clean layer and dedicated file

This creates an isolated layer around the container FD code and everything under it, including the VFIO iommu drivers. All this code is placed into container.c, along with the "struct vfio_container" to compartmentalize it.

Future patches will provide an iommufd based layer that gives the same API as the container layer and choose which layer to go to based on how userspace operates.

The patches continue to split up existing functions and finally the last patch just moves every function that is a "container" function to the new file and creates the global symbols to link them together.

Cross-file container functions are prefixed with vfio_container_* for clarity.

The last patch can be defered and queued during the merge window to manage conflicts. The earlier patches should be fine immediately conflicts wise.

This is the last big series I have to enable basic iommufd functionality. As part of the iommufd series the entire container.c becomes conditionally compiled:

https://github.com/jgunthorpe/linux/commits/vfio_iommufd

This applies on top of the prior two series:
Break up ioctl dispatch functions to one function per ioctl Remove private items from linux/vfio_pci_core.h

Jason Gunthorpe (8):
vfio: Add header guards and includes to drivers/vfio/vfio.h
vfio: Rename __vfio_group_unset_container()
vfio: Split the container logic into vfio_container_attach_group()
vfio: Remove #ifdefs around CONFIG_VFIO_NOIOMMU
vfio: Split out container code from the init/cleanup functions
vfio: Rename vfio_ioctl_check_extension()
vfio: Split the register_device ops call into functions
vfio: Move container code into drivers/vfio/container.c

 drivers/vfio/Makefile    |   1 +
 drivers/vfio/container.c | 680 +++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio.h      |  57 ++++
 drivers/vfio/vfio_main.c | 709 ++-------------------------------------
 4 files changed, 766 insertions(+), 681 deletions(-)
 create mode 100644 drivers/vfio/container.c

base-commit: 456bc2e671af7852f8a028c8069906b6b6fa1ff9

Signature · 1 line
2.37.2
#1 lore raw

PATCH 1/8 vfio: Add header guards and includes to drivers/vfio/vfio.h

As is normal for headers.

---
 drivers/vfio/vfio.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/vfio/vfio.h +10 -0
index 503bea6c843d56..093784f1dea7a9 100644
@@ -3,6 +3,14 @@
  * Copyright (C) 2012 Red Hat, Inc.  All rights reserved.
  *     Author: Alex Williamson <[email protected]>
  */
+#ifndef __VFIO_VFIO_H__
+#define __VFIO_VFIO_H__
+
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/module.h>
+
+struct iommu_group;
 
 enum vfio_group_type {
 	/*
@@ -69,3 +77,5 @@ struct vfio_iommu_driver_ops {
 
 int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
+
+#endif
Signature · 1 line
2.37.2
#2 lore raw

RE: [PATCH 1/8] vfio: Add header guards and includes to drivers/vfio/vfio.h

Quoted 15 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

As is normal for headers.

---
 drivers/vfio/vfio.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/vfio/vfio.h +0 -0
index 503bea6c843d56..093784f1dea7a9 100644

drivers/vfio/vfio.h line 11

Quoted 9 lines
drivers/vfio/vfio.h +6 -0
@@ -3,6 +3,14 @@
  * Copyright (C) 2012 Red Hat, Inc.  All rights reserved.
  *     Author: Alex Williamson <[email protected]>
  */
+#ifndef __VFIO_VFIO_H__
+#define __VFIO_VFIO_H__
+
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/module.h>

Curious what is the criteria for which header inclusions should be placed here. If it is for everything required by the definitions in this file then the list is not complete, e.g. <linux/iommu.h> is obviously missing.

btw while they are moved here the inclusions in vfio_main.c are not removed in patch8.

Quoted 14 lines
drivers/vfio/vfio.h +2 -0
+
+struct iommu_group;
 
 enum vfio_group_type {
 	/*
@@ -69,3 +77,5 @@ struct vfio_iommu_driver_ops {
 
 int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops

*ops);

drivers/vfio/vfio.h +2 -1
+
+#endif
--

2.37.2

#3 lore raw

Re: [PATCH 1/8] vfio: Add header guards and includes to drivers/vfio/vfio.h

Quoted 11 lines · Kevin
Quoted 11 lines
drivers/vfio/vfio.h +3 -0
index 503bea6c843d56..093784f1dea7a9 100644
@@ -3,6 +3,14 @@
  * Copyright (C) 2012 Red Hat, Inc.  All rights reserved.
  *     Author: Alex Williamson <[email protected]>
  */
+#ifndef __VFIO_VFIO_H__
+#define __VFIO_VFIO_H__
+
Quoted 8 lines

drivers/vfio/vfio.h line 11

Quoted 3 lines
drivers/vfio/vfio.h +3 -0
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/module.h>

Curious what is the criteria for which header inclusions should be placed here. If it is for everything required by the definitions in this file then the list is not complete, e.g. <linux/iommu.h> is obviously missing.

It isn't missing:

$ clang-14 -Wp,-MMD,drivers/vfio/.vfio_main.o.d -nostdinc -I../arch/x86/include -I./arch/x86/include/generated -I../include -I./include -I../arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I../include/uapi -I./include/generated/uapi -include ../include/linux/compiler-version.h -include ../include/linux/kconfig.h -include ../include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -fmacro-prefix-map=../= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=unknown-warning-option -Werror=ignored-optimization-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fno-stack-protector -Wimplicit-fallthrough -Wno-gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -fno-stack-clash-protection -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -fno-strict-overflow -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access -I ../drivers/vfio -I ./drivers/vfio  -DMODULE  -DKBUILD_BASENAME='"vfio_main"' -DKBUILD_MODNAME='"vfio"' -D__KBUILD_MODNAME=kmod_vfio -c -o /tmp/jnk.o ../drivers/vfio/vfio.h
[no error]

The criteria I like to use is if the header is able to compile stand-alone.

Quoted 2 lines

btw while they are moved here the inclusions in vfio_main.c are not removed in patch8.

? I'm not sure I understand this

Jason

#4 lore raw

Re: [PATCH 1/8] vfio: Add header guards and includes to drivers/vfio/vfio.h

Quoted 22 lines · On Wed, 31 Aug 2022 12:36:02 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 11 lines · Kevin
Quoted 11 lines
drivers/vfio/vfio.h +3 -0
index 503bea6c843d56..093784f1dea7a9 100644
@@ -3,6 +3,14 @@
  * Copyright (C) 2012 Red Hat, Inc.  All rights reserved.
  *     Author: Alex Williamson <[email protected]>
  */
+#ifndef __VFIO_VFIO_H__
+#define __VFIO_VFIO_H__
+
Quoted 8 lines

drivers/vfio/vfio.h line 11

Quoted 3 lines
drivers/vfio/vfio.h +3 -0
+#include <linux/device.h>
+#include <linux/cdev.h>
+#include <linux/module.h>  

Curious what is the criteria for which header inclusions should be placed here. If it is for everything required by the definitions in this file then the list is not complete, e.g. <linux/iommu.h> is obviously missing.

Quoted 3 lines

It isn't missing:

$ clang-14 -Wp,-MMD,drivers/vfio/.vfio_main.o.d -nostdinc -I../arch/x86/include -I./arch/x86/include/generated -I../include -I./include -I../arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I../include/uapi -I./include/generated/uapi -include ../include/linux/compiler-version.h -include ../include/linux/kconfig.h -include ../include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -fmacro-prefix-map=../= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=unknown-warning-option -Werror=ignored-optimization-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-u

nwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fno-stack-protector -Wimplicit-fallthrough -Wno-gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -fno-stack-clash-protection -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -fno-strict-overflow -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access -I ../drivers/vfio -I ./drivers/vfio -DMODULE -DKBUILD_BASENAME='"vfio_main"' -DKBUILD_MODNAME='"vfio"' -D__KBUILD_MODNAME=kmod_vfio -c -o /tmp/jnk.o ../drivers/vfio/vfio.h

Quoted 4 lines

[no error]

The criteria I like to use is if the header is able to compile stand-alone.

Is this stream of consciousness or is there some tooling for this? ;)

Quoted 4 lines
Quoted 2 lines

btw while they are moved here the inclusions in vfio_main.c are not removed in patch8.

? I'm not sure I understand this

I think Kevin is asking why these includes were not also removed from vfio_main.c when adding them to vfio.h. Thanks,

Alex

#5 lore raw

Re: [PATCH 1/8] vfio: Add header guards and includes to drivers/vfio/vfio.h

Quoted 4 lines · Alex Williamson
Quoted 2 lines

The criteria I like to use is if the header is able to compile stand-alone.

Is this stream of consciousness or is there some tooling for this? ;)

In my case I'm using clangd https://clangd.llvm.org/ in the editor, which checks header files for self-consistency.

But there is also https://include-what-you-use.org/ (though I have never tried it)

But the above wack of text is just the normal compiler invocation of vfio_main.c with main.c replaced by the header.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

btw while they are moved here the inclusions in vfio_main.c are not removed in patch8.

? I'm not sure I understand this

I think Kevin is asking why these includes were not also removed from vfio_main.c when adding them to vfio.h. Thanks,

Oh, I am actually unclear what is policy/preference/consensus in that area.

I know a strong camp is to avoid implicit includes, so the duplicated includes are welcomed. include-what-you-use for instance is that philosophy.

Do you have a preference?

Jason

#6 lore raw

RE: [PATCH 1/8] vfio: Add header guards and includes to drivers/vfio/vfio.h

Quoted 10 lines

From: Jason Gunthorpe <[email protected]>
Sent: Thursday, September 1, 2022 2:25 AM

Quoted 4 lines · Alex Williamson
Quoted 2 lines

The criteria I like to use is if the header is able to compile stand-alone.

Is this stream of consciousness or is there some tooling for this? ;)

Quoted 8 lines

In my case I'm using clangd https://clangd.llvm.org/ in the editor, which checks header files for self-consistency.

But there is also https://include-what-you-use.org/ (though I have never tried it)

But the above wack of text is just the normal compiler invocation of vfio_main.c with main.c replaced by the header.

Thanks. This is a good learning.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

btw while they are moved here the inclusions in vfio_main.c are not removed in patch8.

? I'm not sure I understand this

I think Kevin is asking why these includes were not also removed from vfio_main.c when adding them to vfio.h. Thanks,

yes

Quoted 3 lines

Oh, I am actually unclear what is policy/preference/consensus in that area.

me either. That is why I raised this open in case there is already a consensus in place. 😊

Quoted 8 lines

I know a strong camp is to avoid implicit includes, so the duplicated includes are welcomed. include-what-you-use for instance is that philosophy.

Do you have a preference?

Jason

#7 lore raw

PATCH 2/8 vfio: Rename __vfio_group_unset_container()

In reply to Jason Gunthorpe

To vfio_container_detatch_group(). This function is really a container function.

Fold the WARN_ON() into it as a precondition assertion.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/vfio/vfio_main.c +5 -6
index bfa6119ba47337..e145c87f208f3a 100644
@@ -928,12 +928,13 @@ static const struct file_operations vfio_fops = {
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
-static void __vfio_group_unset_container(struct vfio_group *group)
+static void vfio_container_detatch_group(struct vfio_group *group)
 {
 	struct vfio_container *container = group->container;
 	struct vfio_iommu_driver *driver;
 
 	lockdep_assert_held_write(&group->group_rwsem);
+	WARN_ON(group->container_users != 1);
 
 	down_write(&container->group_lock);
 
@@ -976,7 +977,7 @@ static int vfio_group_ioctl_unset_container(struct vfio_group *group)
 		return -EINVAL;
 	if (group->container_users != 1)
 		return -EBUSY;
-	__vfio_group_unset_container(group);
+	vfio_container_detatch_group(group);
 	return 0;
 }
 
@@ -1329,10 +1330,8 @@ static int vfio_group_fops_release(struct inode *inode, struct file *filep)
 	 * is only called when there are no open devices.
 	 */
 	WARN_ON(group->notifier.head);
-	if (group->container) {
-		WARN_ON(group->container_users != 1);
-		__vfio_group_unset_container(group);
-	}
+	if (group->container)
+		vfio_container_detatch_group(group);
 	group->opened_file = NULL;
 	up_write(&group->group_rwsem);
 
Signature · 1 line
2.37.2
#8 lore raw

RE: [PATCH 2/8] vfio: Rename __vfio_group_unset_container()

Quoted 21 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

To vfio_container_detatch_group(). This function is really a container function.

Fold the WARN_ON() into it as a precondition assertion.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/vfio/vfio_main.c +0 -0
index bfa6119ba47337..e145c87f208f3a 100644

drivers/vfio/vfio_main.c static const struct file_operations vfio_fops = { · line 931

Quoted 5 lines
drivers/vfio/vfio_main.c +1 -1
@@ -928,12 +928,13 @@ static const struct file_operations vfio_fops = {
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
-static void __vfio_group_unset_container(struct vfio_group *group)
+static void vfio_container_detatch_group(struct vfio_group *group)

s/detatch/detach/

Given it's a vfio_container function is it better to have a container pointer as the first parameter, i.e.:

static void vfio_container_detatch_group(struct vfio_container *container,
struct vfio_group *group)

ditto for patch7:

+static void vfio_container_register_device(struct vfio_device *device)

#9 lore raw

Re: [PATCH 2/8] vfio: Rename __vfio_group_unset_container()

Quoted 16 lines · Kevin
Quoted 16 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

To vfio_container_detatch_group(). This function is really a container function.

Fold the WARN_ON() into it as a precondition assertion.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
Quoted 12 lines

drivers/vfio/vfio_main.c static const struct file_operations vfio_fops = { · line 931

Quoted 10 lines
drivers/vfio/vfio_main.c +1 -1
index bfa6119ba47337..e145c87f208f3a 100644
@@ -928,12 +928,13 @@ static const struct file_operations vfio_fops = {
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
-static void __vfio_group_unset_container(struct vfio_group *group)
+static void vfio_container_detatch_group(struct vfio_group *group)

s/detatch/detach/

Oops

Quoted 5 lines

Given it's a vfio_container function is it better to have a container pointer as the first parameter, i.e.:

static void vfio_container_detatch_group(struct vfio_container *container,
struct vfio_group *group)

Ah, I'm not so sure, it seems weird to make the caller do group->container then pass the group in as well.

This call assumes the container is the container of the group, it doesn't work in situations where that isn't true.

It is kind of weird layering in a way, arguably we should have the current group stored in the container if we want things to work that way, but that is a big change and not that wortwhile I think.

Patch 7 is pretty much the same, it doesn't work right unless the container and device are already matched

Thanks,
Jason

Reviewed-by
#10 lore raw

RE: [PATCH 2/8] vfio: Rename __vfio_group_unset_container()

Quoted 38 lines

From: Jason Gunthorpe <[email protected]>
Sent: Friday, September 2, 2022 8:29 AM

Quoted 16 lines · Kevin
Quoted 16 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

To vfio_container_detatch_group(). This function is really a container function.

Fold the WARN_ON() into it as a precondition assertion.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
Quoted 12 lines

drivers/vfio/vfio_main.c static const struct file_operations vfio_fops = { · line 931

Quoted 10 lines
drivers/vfio/vfio_main.c +1 -1
index bfa6119ba47337..e145c87f208f3a 100644
@@ -928,12 +928,13 @@ static const struct file_operations vfio_fops = {
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
-static void __vfio_group_unset_container(struct vfio_group *group)
+static void vfio_container_detatch_group(struct vfio_group *group)

s/detatch/detach/

Oops

Quoted 3 lines

Given it's a vfio_container function is it better to have a container pointer as the first parameter, i.e.:

Quoted 8 lines
Quoted 2 lines

static void vfio_container_detatch_group(struct vfio_container *container,
struct vfio_group *group)

Ah, I'm not so sure, it seems weird to make the caller do group->container then pass the group in as well.

This call assumes the container is the container of the group, it doesn't work in situations where that isn't true.

Yes, this is a valid interpretation on doing this way.

Other places e.g. iommu_detach_group(domain, group) go the other way even if internally domain is not used at all. I kind of leave that pattern in mind thus raised this comment. But not a strong opinion.

Quoted 8 lines

It is kind of weird layering in a way, arguably we should have the current group stored in the container if we want things to work that way, but that is a big change and not that wortwhile I think.

Patch 7 is pretty much the same, it doesn't work right unless the container and device are already matched

If Alex won't have a different preference and with the typo fixed,

#11 lore raw

Re: [PATCH 2/8] vfio: Rename __vfio_group_unset_container()

Quoted 58 lines · Kevin
Quoted 38 lines

From: Jason Gunthorpe <[email protected]>
Sent: Friday, September 2, 2022 8:29 AM

Quoted 16 lines · Kevin
Quoted 16 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

To vfio_container_detatch_group(). This function is really a container function.

Fold the WARN_ON() into it as a precondition assertion.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
Quoted 12 lines

drivers/vfio/vfio_main.c static const struct file_operations vfio_fops = { · line 931

Quoted 10 lines
drivers/vfio/vfio_main.c +1 -1
index bfa6119ba47337..e145c87f208f3a 100644
@@ -928,12 +928,13 @@ static const struct file_operations vfio_fops = {
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
-static void __vfio_group_unset_container(struct vfio_group *group)
+static void vfio_container_detatch_group(struct vfio_group *group)  

s/detatch/detach/

Oops

Quoted 3 lines

Given it's a vfio_container function is it better to have a container pointer as the first parameter, i.e.:

Quoted 8 lines
Quoted 2 lines

static void vfio_container_detatch_group(struct vfio_container *container,
struct vfio_group *group)

Ah, I'm not so sure, it seems weird to make the caller do group->container then pass the group in as well.

This call assumes the container is the container of the group, it doesn't work in situations where that isn't true.

Yes, this is a valid interpretation on doing this way.

Other places e.g. iommu_detach_group(domain, group) go the other way even if internally domain is not used at all. I kind of leave that pattern in mind thus raised this comment. But not a strong opinion.

Quoted 5 lines

It is kind of weird layering in a way, arguably we should have the current group stored in the container if we want things to work that way, but that is a big change and not that wortwhile I think.

Quoted 5 lines
Quoted 3 lines

Patch 7 is pretty much the same, it doesn't work right unless the container and device are already matched

If Alex won't have a different preference and with the typo fixed,

I don't get it, if a group is detaching itself from a container, why isn't it vfio_group_detach_container(). Given our naming scheme, vfio_container_detach_group() absolutely implies the args Kevin suggests, even if they're redundant. vfio_foo_* functions should operate on a a vfio_foo object. Thanks,

Alex

#12 lore raw

Re: [PATCH 2/8] vfio: Rename __vfio_group_unset_container()

Quoted 65 lines · Alex Williamson
Quoted 62 lines · Kevin
Quoted 38 lines

From: Jason Gunthorpe <[email protected]>
Sent: Friday, September 2, 2022 8:29 AM

Quoted 16 lines · Kevin
Quoted 16 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

To vfio_container_detatch_group(). This function is really a container function.

Fold the WARN_ON() into it as a precondition assertion.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
Quoted 12 lines

drivers/vfio/vfio_main.c static const struct file_operations vfio_fops = { · line 931

Quoted 10 lines
drivers/vfio/vfio_main.c +1 -1
index bfa6119ba47337..e145c87f208f3a 100644
@@ -928,12 +928,13 @@ static const struct file_operations vfio_fops = {
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
-static void __vfio_group_unset_container(struct vfio_group *group)
+static void vfio_container_detatch_group(struct vfio_group *group)  

s/detatch/detach/

Oops

Quoted 3 lines

Given it's a vfio_container function is it better to have a container pointer as the first parameter, i.e.:

Quoted 8 lines
Quoted 2 lines

static void vfio_container_detatch_group(struct vfio_container *container,
struct vfio_group *group)

Ah, I'm not so sure, it seems weird to make the caller do group->container then pass the group in as well.

This call assumes the container is the container of the group, it doesn't work in situations where that isn't true.

Yes, this is a valid interpretation on doing this way.

Other places e.g. iommu_detach_group(domain, group) go the other way even if internally domain is not used at all. I kind of leave that pattern in mind thus raised this comment. But not a strong opinion.

Quoted 8 lines

It is kind of weird layering in a way, arguably we should have the current group stored in the container if we want things to work that way, but that is a big change and not that wortwhile I think.

Patch 7 is pretty much the same, it doesn't work right unless the container and device are already matched

Quoted 7 lines
Quoted 1 line

If Alex won't have a different preference and with the typo fixed,

I don't get it, if a group is detaching itself from a container, why isn't it vfio_group_detach_container(). Given our naming scheme, vfio_container_detach_group() absolutely implies the args Kevin suggests, even if they're redundant. vfio_foo_* functions should operate on a a vfio_foo object.

Look at the overall picture. This series moves struct vfio_container into a .c file and then pulls all of the code that relies on it into the c file too.

With our current function layout that results in these cut points:

struct vfio_container *vfio_container_from_file(struct file *filep); int vfio_container_use(struct vfio_group *group);
void vfio_container_unuse(struct vfio_group *group);
int vfio_container_attach_group(struct vfio_container *container,

				struct vfio_group *group);

void vfio_container_detach_group(struct vfio_group *group); void vfio_container_register_device(struct vfio_device *device); void vfio_container_unregister_device(struct vfio_device *device); long vfio_container_ioctl_check_extension(struct vfio_container *container,

					  unsigned long arg);
int vfio_container_pin_pages(struct vfio_device *device, dma_addr_t iova,
			     int npage, int prot, struct page **pages);
void vfio_container_unpin_pages(struct vfio_device *device, dma_addr_t iova,
				int npage);
int vfio_container_dma_rw(struct vfio_device *device, dma_addr_t iova,
			  void *data, size_t len, bool write);

Notice almost none of them use container as a function argument. The container is always implied by another object that is linked to the container.

Yet these are undeniably all container functions because they are only necessary when the container code is actually being used. Every one of this functions is bypassed if iommmufd is used. I even have a patch that just replaces them all with nops if container and type 1 is compiled out.

So, this presents two possiblities for naming. Either we call everything in container.c vfio_container because it is primarily related to the container.c *system*, or we label each function according to OOP via the first argument type (vfio_XX_YY_container perhaps) and still place them in container.c.

Keep in mind I have plans to see a group.c and rename vfio_main.c to device.c, so having a vfio_group or vfio_device function in container.c seems to get confusing.

In other words, I prefer to think of the group of functions above as the exported API of the vfio container system (ie container.c) - not in terms of an OOP modeling of a vfio_container object.

Jason

#13 lore raw

RE: [PATCH 2/8] vfio: Rename __vfio_group_unset_container()

Quoted 122 lines

From: Jason Gunthorpe <[email protected]>
Sent: Saturday, September 3, 2022 12:25 AM

Quoted 11 lines · Alex Williamson
Quoted 8 lines · Kevin
Quoted 8 lines

From: Jason Gunthorpe <[email protected]>
Sent: Friday, September 2, 2022 8:29 AM

Quoted 4 lines · Kevin
Quoted 4 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

To vfio_container_detatch_group(). This function is really a

container

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

function.

Fold the WARN_ON() into it as a precondition assertion.

A following patch will move the vfio_container functions to their

own .c

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

file.

---
 drivers/vfio/vfio_main.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/vfio/vfio_main.c static const struct file_operations

Quoted 5 lines
drivers/vfio/vfio_main.c +0 -0
index bfa6119ba47337..e145c87f208f3a 100644
@@ -928,12 +928,13 @@ static const struct file_operations

vfio_fops = {

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines

drivers/vfio/vfio_main.c static const struct file_operations vfio_fops = { · line 931

Quoted 5 lines
drivers/vfio/vfio_main.c +1 -1
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
-static void __vfio_group_unset_container(struct vfio_group *group)
+static void vfio_container_detatch_group(struct vfio_group *group)

s/detatch/detach/

Oops

Quoted 1 line

Given it's a vfio_container function is it better to have a container

pointer

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

as the first parameter, i.e.:

static void vfio_container_detatch_group(struct vfio_container

*container,

Quoted 23 lines
Quoted 23 lines
Quoted 7 lines
Quoted 1 line

struct vfio_group *group)

Ah, I'm not so sure, it seems weird to make the caller do group->container then pass the group in as well.

This call assumes the container is the container of the group, it doesn't work in situations where that isn't true.

Yes, this is a valid interpretation on doing this way.

Other places e.g. iommu_detach_group(domain, group) go the other way even if internally domain is not used at all. I kind of leave that pattern in mind thus raised this comment. But not a strong opinion.

Quoted 8 lines

It is kind of weird layering in a way, arguably we should have the current group stored in the container if we want things to work that way, but that is a big change and not that wortwhile I think.

Patch 7 is pretty much the same, it doesn't work right unless the container and device are already matched

Quoted 7 lines
Quoted 1 line

If Alex won't have a different preference and with the typo fixed,

I don't get it, if a group is detaching itself from a container, why isn't it vfio_group_detach_container(). Given our naming scheme, vfio_container_detach_group() absolutely implies the args Kevin suggests, even if they're redundant. vfio_foo_* functions should operate on a a vfio_foo object.

Look at the overall picture. This series moves struct vfio_container into a .c file and then pulls all of the code that relies on it into the c file too.

With our current function layout that results in these cut points:

struct vfio_container *vfio_container_from_file(struct file *filep); int vfio_container_use(struct vfio_group *group);
void vfio_container_unuse(struct vfio_group *group);
int vfio_container_attach_group(struct vfio_container *container,

				struct vfio_group *group);

void vfio_container_detach_group(struct vfio_group *group); void vfio_container_register_device(struct vfio_device *device); void vfio_container_unregister_device(struct vfio_device *device); long vfio_container_ioctl_check_extension(struct vfio_container *container,

					  unsigned long arg);
int vfio_container_pin_pages(struct vfio_device *device, dma_addr_t iova,
			     int npage, int prot, struct page **pages);
void vfio_container_unpin_pages(struct vfio_device *device, dma_addr_t
iova,
				int npage);
int vfio_container_dma_rw(struct vfio_device *device, dma_addr_t iova,
			  void *data, size_t len, bool write);

Notice almost none of them use container as a function argument. The container is always implied by another object that is linked to the container.

Yet these are undeniably all container functions because they are only necessary when the container code is actually being used. Every one of this functions is bypassed if iommmufd is used. I even have a patch that just replaces them all with nops if container and type 1 is compiled out.

So, this presents two possiblities for naming. Either we call everything in container.c vfio_container because it is primarily related to the container.c *system*, or we label each function according to OOP via the first argument type (vfio_XX_YY_container perhaps) and still place them in container.c.

Quoted 3 lines

Keep in mind I have plans to see a group.c and rename vfio_main.c to device.c, so having a vfio_group or vfio_device function in container.c seems to get confusing.

IMHO I don't see a big difference between two naming options if the first parameter is kept as group or device object.

Quoted 5 lines

In other words, I prefer to think of the group of functions above as the exported API of the vfio container system (ie container.c) - not in terms of an OOP modeling of a vfio_container object.

After reading above IMHO the OOP modeling wins a bit as far as readability is concerned. Probably just my personal preference but having most vfio_maintaier_xxx() helpers w/o explicitly providing a vfio_maintainer object does affect my reading of related code.

At least vfio_XX_YY_container makes me feel better if we want to avoid duplicating a vfio_maintainer object, e.g.:

struct vfio_container *vfio_container_from_file(struct file *filep); int vfio_group_use_container(struct vfio_group *group); void vfio_group_unuse_container(struct vfio_group *group); int vfio_group_attach_container(struct vfio_group *group,

				vfio_container *container);

void vfio_group_detach_container(struct vfio_group *group); void vfio_device_register_container(struct vfio_device *device); void vfio_device_unregister_container(struct vfio_device *device); long vfio_container_ioctl_check_extension(struct vfio_container *container,

					  unsigned long arg);
int vfio_pin_pages_container(struct vfio_device *device, dma_addr_t iova,
			     int npage, int prot, struct page **pages);
void vfio_unpin_pages_container(struct vfio_device *device, dma_addr_t
iova,
				int npage);
int vfio_dma_rw_container(struct vfio_device *device, dma_addr_t iova,
			  void *data, size_t len, bool write);

They are all container related. So although the first parameter is not container we put them in container.c as the last word in the function name is container.

Thanks
Kevin

#14 lore raw

Re: [PATCH 2/8] vfio: Rename __vfio_group_unset_container()

Quoted 161 lines · Kevin
Quoted 122 lines

From: Jason Gunthorpe <[email protected]>
Sent: Saturday, September 3, 2022 12:25 AM

Quoted 11 lines · Alex Williamson
Quoted 8 lines · Kevin
Quoted 8 lines

From: Jason Gunthorpe <[email protected]>
Sent: Friday, September 2, 2022 8:29 AM

Quoted 4 lines · Kevin
Quoted 4 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

To vfio_container_detatch_group(). This function is really a

container

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

function.

Fold the WARN_ON() into it as a precondition assertion.

A following patch will move the vfio_container functions to their

own .c

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

file.

---
 drivers/vfio/vfio_main.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/vfio/vfio_main.c static const struct file_operations

Quoted 5 lines
drivers/vfio/vfio_main.c +0 -0
index bfa6119ba47337..e145c87f208f3a 100644
@@ -928,12 +928,13 @@ static const struct file_operations  

vfio_fops = {

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines

drivers/vfio/vfio_main.c static const struct file_operations · line 931

Quoted 5 lines
drivers/vfio/vfio_main.c +1 -1
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
-static void __vfio_group_unset_container(struct vfio_group *group)
+static void vfio_container_detatch_group(struct vfio_group *group)  

s/detatch/detach/

Oops

Quoted 1 line

Given it's a vfio_container function is it better to have a container

pointer

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

as the first parameter, i.e.:

static void vfio_container_detatch_group(struct vfio_container

*container,

Quoted 23 lines
Quoted 23 lines
Quoted 7 lines
Quoted 1 line

struct vfio_group *group)

Ah, I'm not so sure, it seems weird to make the caller do group->container then pass the group in as well.

This call assumes the container is the container of the group, it doesn't work in situations where that isn't true.

Yes, this is a valid interpretation on doing this way.

Other places e.g. iommu_detach_group(domain, group) go the other way even if internally domain is not used at all. I kind of leave that pattern in mind thus raised this comment. But not a strong opinion.

Quoted 8 lines

It is kind of weird layering in a way, arguably we should have the current group stored in the container if we want things to work that way, but that is a big change and not that wortwhile I think.

Patch 7 is pretty much the same, it doesn't work right unless the container and device are already matched

Quoted 7 lines
Quoted 1 line

If Alex won't have a different preference and with the typo fixed,

I don't get it, if a group is detaching itself from a container, why isn't it vfio_group_detach_container(). Given our naming scheme, vfio_container_detach_group() absolutely implies the args Kevin suggests, even if they're redundant. vfio_foo_* functions should operate on a a vfio_foo object.

Look at the overall picture. This series moves struct vfio_container into a .c file and then pulls all of the code that relies on it into the c file too.

With our current function layout that results in these cut points:

struct vfio_container *vfio_container_from_file(struct file *filep); int vfio_container_use(struct vfio_group *group);
void vfio_container_unuse(struct vfio_group *group);
int vfio_container_attach_group(struct vfio_container *container,

				struct vfio_group *group);

void vfio_container_detach_group(struct vfio_group *group); void vfio_container_register_device(struct vfio_device *device); void vfio_container_unregister_device(struct vfio_device *device); long vfio_container_ioctl_check_extension(struct vfio_container *container,

					  unsigned long arg);
int vfio_container_pin_pages(struct vfio_device *device, dma_addr_t iova,
			     int npage, int prot, struct page **pages);
void vfio_container_unpin_pages(struct vfio_device *device, dma_addr_t
iova,
				int npage);
int vfio_container_dma_rw(struct vfio_device *device, dma_addr_t iova,
			  void *data, size_t len, bool write);

Notice almost none of them use container as a function argument. The container is always implied by another object that is linked to the container.

Yet these are undeniably all container functions because they are only necessary when the container code is actually being used. Every one of this functions is bypassed if iommmufd is used. I even have a patch that just replaces them all with nops if container and type 1 is compiled out.

So, this presents two possiblities for naming. Either we call everything in container.c vfio_container because it is primarily related to the container.c *system*, or we label each function according to OOP via the first argument type (vfio_XX_YY_container perhaps) and still place them in container.c.

Quoted 3 lines

Keep in mind I have plans to see a group.c and rename vfio_main.c to device.c, so having a vfio_group or vfio_device function in container.c seems to get confusing.

IMHO I don't see a big difference between two naming options if the first parameter is kept as group or device object.

Quoted 5 lines

In other words, I prefer to think of the group of functions above as the exported API of the vfio container system (ie container.c) - not in terms of an OOP modeling of a vfio_container object.

After reading above IMHO the OOP modeling wins a bit as far as readability is concerned. Probably just my personal preference but having most vfio_maintaier_xxx() helpers w/o explicitly providing a vfio_maintainer object does affect my reading of related code.

At least vfio_XX_YY_container makes me feel better if we want to avoid duplicating a vfio_maintainer object, e.g.:

struct vfio_container *vfio_container_from_file(struct file *filep); int vfio_group_use_container(struct vfio_group *group); void vfio_group_unuse_container(struct vfio_group *group); int vfio_group_attach_container(struct vfio_group *group,

				vfio_container *container);

void vfio_group_detach_container(struct vfio_group *group); void vfio_device_register_container(struct vfio_device *device); void vfio_device_unregister_container(struct vfio_device *device); long vfio_container_ioctl_check_extension(struct vfio_container *container,

					  unsigned long arg);
int vfio_pin_pages_container(struct vfio_device *device, dma_addr_t iova,
			     int npage, int prot, struct page **pages);
void vfio_unpin_pages_container(struct vfio_device *device, dma_addr_t
iova,
				int npage);
int vfio_dma_rw_container(struct vfio_device *device, dma_addr_t iova,
			  void *data, size_t len, bool write);
Quoted 3 lines

They are all container related. So although the first parameter is not container we put them in container.c as the last word in the function name is container.

I might refine these to:

struct vfio_container *vfio_container_from_file(struct file *filep);

int vfio_group_use_container(struct vfio_group *group); void vfio_group_unuse_container(struct vfio_group *group);

int vfio_container_attach_group(struct vfio_container *container,

				struct vfio_group *group);

void vfio_group_detach_container(struct vfio_group *group);

void vfio_device_container_register(struct vfio_device *device); void vfio_device_container_unregister(struct vfio_device *device);

long vfio_container_ioctl_check_extension(struct vfio_container *container,
unsigned long arg);

int vfio_device_pin_container_pages(struct vfio_device *device, dma_addr_t iova,

				    int npage, int prot, struct page **pages);
void vfio_device_unpin_container_pages(struct vfio_device *device, dma_addr_t
				       iova, int npage);

int vfio_device_dma_rw_container(struct vfio_device *device, dma_addr_t iova,

				 void *data, size_t len, bool write);

Overall, I'm still not really on board with sacrificing a "the name tells you how to use it" API in order to break down devices, groups, and containers into their own subsystems. We should not only consider the logical location of the function, but the usability and intuitiveness of the API.

Are we necessarily finding the right splits here? The {un}use, {un}pin, and dma_rw in particular seem like they could be decomposed further. For example a vfio_group intermediary that calls vfio_container_{un}use() with a vfio_container object. Or a vfio_device intermediary that can call vfio_container_ functions for {un}pin/rw, also with a container object. Thanks,

Alex

#15 lore raw

Re: [PATCH 2/8] vfio: Rename __vfio_group_unset_container()

Quoted 10 lines · Alex Williamson

I might refine these to:

struct vfio_container *vfio_container_from_file(struct file *filep);

int vfio_group_use_container(struct vfio_group *group); void vfio_group_unuse_container(struct vfio_group *group);

int vfio_container_attach_group(struct vfio_container *container,

				struct vfio_group *group);

void vfio_group_detach_container(struct vfio_group *group);

IMHO it is weird to sacrifice the clear pair'd naming, keep the group first?

Quoted 5 lines

void vfio_device_container_register(struct vfio_device *device); void vfio_device_container_unregister(struct vfio_device *device);

long vfio_container_ioctl_check_extension(struct vfio_container *container,
unsigned long arg);

I'm fine with all of these, if you like it I'll change it - it is a lot of work to change the names and rebase everything so please lets all agree first.

Quoted 7 lines

int vfio_device_pin_container_pages(struct vfio_device *device, dma_addr_t iova,

				    int npage, int prot, struct page **pages);
void vfio_device_unpin_container_pages(struct vfio_device *device, dma_addr_t
				       iova, int npage);

int vfio_device_dma_rw_container(struct vfio_device *device, dma_addr_t iova,

				 void *data, size_t len, bool write);

These are exported symbols and I don't want to mess with them. The fact they are in container.c is temporary artifact, as they do touch the struct vfio_container. The iommufd series puts a wrapper in main.c and the above can have signatures that only take in a container * - which is what you suggest below. So lets leave them alone here.

Quoted 5 lines

Overall, I'm still not really on board with sacrificing a "the name tells you how to use it" API in order to break down devices, groups, and containers into their own subsystems. We should not only consider the logical location of the function, but the usability and intuitiveness of the API.

Sure, I'm not set on any particular naming scheme.

Quoted 3 lines

Are we necessarily finding the right splits here? The {un}use, {un}pin, and dma_rw in particular seem like they could be decomposed further.

I can't think how to do anything better for {un}use. They are in container.c because the entire container_users mechanism is gone when the container code is gone, even though the mechanism is part of the struct vfio_group. unuse doesn't even touch the vfio_container..

Lets just leave them with their group names and they slightly oddly stay in container.c

Quoted 3 lines

vfio_container_{un}use() with a vfio_container object. Or a vfio_device intermediary that can call vfio_container_ functions for {un}pin/rw, also with a container object. Thanks,

Some of the split is an artifact of how the code is right now. I don't think you'd end up with exactly this interface if you designed it from scratch, but we have what we have, and I don't have a lot of enthusiasm to restructure significantly for the sake of naming..

Thanks,
Jason

#16 lore raw

PATCH 3/8 vfio: Split the container logic into vfio_container_attach_group()

In reply to Jason Gunthorpe

This splits up the ioctl of vfio_group_ioctl_set_container() so it determines the type of file then invokes a type specific attachment function. Future patches will add iommufd to this function as an alternative type.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 78 ++++++++++++++++++++++++----------------
 1 file changed, 48 insertions(+), 30 deletions(-)
drivers/vfio/vfio_main.c +48 -30
index e145c87f208f3a..1108ba53fe5c28 100644
@@ -981,40 +981,29 @@ static int vfio_group_ioctl_unset_container(struct vfio_group *group)
 	return 0;
 }
 
-static int vfio_group_ioctl_set_container(struct vfio_group *group,
-					  int __user *arg)
+static struct vfio_container *vfio_container_from_file(struct file *file)
 {
-	struct fd f;
 	struct vfio_container *container;
-	struct vfio_iommu_driver *driver;
-	int container_fd;
-	int ret = 0;
-
-	if (group->type == VFIO_NO_IOMMU && !capable(CAP_SYS_RAWIO))
-		return -EPERM;
-
-	if (get_user(container_fd, arg))
-		return -EFAULT;
-	if (container_fd < 0)
-		return -EINVAL;
-	f = fdget(container_fd);
-	if (!f.file)
-		return -EBADF;
 
 	/* Sanity check, is this really our fd? */
-	if (f.file->f_op != &vfio_fops) {
-		ret = -EINVAL;
-		goto out_fdput;
-	}
-	container = f.file->private_data;
+	if (file->f_op != &vfio_fops)
+		return NULL;
+
+	container = file->private_data;
 	WARN_ON(!container); /* fget ensures we don't race vfio_release */
+	return container;
+}
 
-	down_write(&group->group_rwsem);
+static int vfio_container_attach_group(struct vfio_group *group,
+				       struct vfio_container *container)
+{
+	struct vfio_iommu_driver *driver;
+	int ret = 0;
 
-	if (group->container || WARN_ON(group->container_users)) {
-		ret = -EINVAL;
-		goto out_unlock_group;
-	}
+	lockdep_assert_held_write(&group->group_rwsem);
+
+	if (group->type == VFIO_NO_IOMMU && !capable(CAP_SYS_RAWIO))
+		return -EPERM;
 
 	down_write(&container->group_lock);
 
@@ -1026,7 +1015,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 	}
 
 	if (group->type == VFIO_IOMMU) {
-		ret = iommu_group_claim_dma_owner(group->iommu_group, f.file);
+		ret = iommu_group_claim_dma_owner(group->iommu_group, group);
 		if (ret)
 			goto out_unlock_container;
 	}
@@ -1054,9 +1043,38 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 
 out_unlock_container:
 	up_write(&container->group_lock);
-out_unlock_group:
+	return ret;
+}
+
+static int vfio_group_ioctl_set_container(struct vfio_group *group,
+					  int __user *arg)
+{
+	struct vfio_container *container;
+	struct fd f;
+	int ret;
+	int fd;
+
+	if (get_user(fd, arg))
+		return -EFAULT;
+
+	f = fdget(fd);
+	if (!f.file)
+		return -EBADF;
+
+	down_write(&group->group_rwsem);
+	if (group->container || WARN_ON(group->container_users)) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+	container = vfio_container_from_file(f.file);
+	ret = -EINVAL;
+	if (container) {
+		ret = vfio_container_attach_group(group, container);
+		goto out_unlock;
+	}
+
+out_unlock:
 	up_write(&group->group_rwsem);
-out_fdput:
 	fdput(f);
 	return ret;
 }
Signature · 1 line
2.37.2
Reviewed-by on v1 3/8
#17 lore raw

RE: [PATCH 3/8] vfio: Split the container logic into vfio_container_attach_group()

drivers/vfio/vfio_main.c static int vfio_group_ioctl_unset_container(struct vfio_group *group) · line 998

Quoted 5 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

drivers/vfio/vfio_main.c +2 -0
+static int vfio_container_attach_group(struct vfio_group *group,
+				       struct vfio_container *container)

exchange the order of parameters.

Otherwise,

#18 lore raw

Re: [PATCH 3/8] vfio: Split the container logic into vfio_container_attach_group()

Quoted 11 lines · Kevin

drivers/vfio/vfio_main.c static int vfio_group_ioctl_unset_container(struct vfio_group *group) · line 998

Quoted 5 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

drivers/vfio/vfio_main.c +2 -0
+static int vfio_container_attach_group(struct vfio_group *group,
+				       struct vfio_container *container)

exchange the order of parameters.

Otherwise,

Done, thanks

Jason

#19 lore raw

PATCH 4/8 vfio: Remove #ifdefs around CONFIG_VFIO_NOIOMMU

In reply to Jason Gunthorpe

This can all be accomplished using typical IS_ENABLED techniques, drop it all.

Also rename the variable to vfio_noiommu so this can be made global in following patches.

---
 drivers/vfio/vfio_main.c | 43 ++++++++++++++++++----------------------
 1 file changed, 19 insertions(+), 24 deletions(-)
drivers/vfio/vfio_main.c +19 -24
index 1108ba53fe5c28..f76cea312028a6 100644
@@ -82,10 +82,12 @@ struct vfio_group {
 };
 
 #ifdef CONFIG_VFIO_NOIOMMU
-static bool noiommu __read_mostly;
+static bool vfio_noiommu __read_mostly;
 module_param_named(enable_unsafe_noiommu_mode,
-		   noiommu, bool, S_IRUGO | S_IWUSR);
+		   vfio_noiommu, bool, S_IRUGO | S_IWUSR);
 MODULE_PARM_DESC(enable_unsafe_noiommu_mode, "Enable UNSAFE, no-IOMMU mode.  This mode provides no device isolation, no DMA translation, no host kernel protection, cannot be used for device assignment to virtual machines, requires RAWIO permissions, and will taint the kernel.  If you do not know what this is for, step away. (default: false)");
+#else
+enum { vfio_noiommu = false };
 #endif
 
 static DEFINE_XARRAY(vfio_device_set_xa);
@@ -162,7 +164,6 @@ static void vfio_release_device_set(struct vfio_device *device)
 	xa_unlock(&vfio_device_set_xa);
 }
 
-#ifdef CONFIG_VFIO_NOIOMMU
 static void *vfio_noiommu_open(unsigned long arg)
 {
 	if (arg != VFIO_NOIOMMU_IOMMU)
@@ -181,7 +182,7 @@ static long vfio_noiommu_ioctl(void *iommu_data,
 			       unsigned int cmd, unsigned long arg)
 {
 	if (cmd == VFIO_CHECK_EXTENSION)
-		return noiommu && (arg == VFIO_NOIOMMU_IOMMU) ? 1 : 0;
+		return vfio_noiommu && (arg == VFIO_NOIOMMU_IOMMU) ? 1 : 0;
 
 	return -ENOTTY;
 }
@@ -211,18 +212,13 @@ static const struct vfio_iommu_driver_ops vfio_noiommu_ops = {
  * Only noiommu containers can use vfio-noiommu and noiommu containers can only
  * use vfio-noiommu.
  */
-static inline bool vfio_iommu_driver_allowed(struct vfio_container *container,
-		const struct vfio_iommu_driver *driver)
+static bool vfio_iommu_driver_allowed(struct vfio_container *container,
+				      const struct vfio_iommu_driver *driver)
 {
+	if (!IS_ENABLED(CONFIG_VFIO_NOIOMMU))
+		return true;
 	return container->noiommu == (driver->ops == &vfio_noiommu_ops);
 }
-#else
-static inline bool vfio_iommu_driver_allowed(struct vfio_container *container,
-		const struct vfio_iommu_driver *driver)
-{
-	return true;
-}
-#endif /* CONFIG_VFIO_NOIOMMU */
 
 /*
  * IOMMU driver registration
@@ -535,8 +531,7 @@ static struct vfio_group *vfio_group_find_or_alloc(struct device *dev)
 	struct vfio_group *group;
 
 	iommu_group = iommu_group_get(dev);
-#ifdef CONFIG_VFIO_NOIOMMU
-	if (!iommu_group && noiommu) {
+	if (!iommu_group && vfio_noiommu) {
 		/*
 		 * With noiommu enabled, create an IOMMU group for devices that
 		 * don't already have one, implying no IOMMU hardware/driver
@@ -550,7 +545,7 @@ static struct vfio_group *vfio_group_find_or_alloc(struct device *dev)
 		}
 		return group;
 	}
-#endif
+
 	if (!iommu_group)
 		return ERR_PTR(-EINVAL);
 
@@ -2101,11 +2096,11 @@ static int __init vfio_init(void)
 	if (ret)
 		goto err_alloc_chrdev;
 
-#ifdef CONFIG_VFIO_NOIOMMU
-	ret = vfio_register_iommu_driver(&vfio_noiommu_ops);
-#endif
-	if (ret)
-		goto err_driver_register;
+	if (IS_ENABLED(CONFIG_VFIO_NOIOMMU)) {
+		ret = vfio_register_iommu_driver(&vfio_noiommu_ops);
+		if (ret)
+			goto err_driver_register;
+	}
 
 	pr_info(DRIVER_DESC " version: " DRIVER_VERSION "\n");
 	return 0;
@@ -2124,9 +2119,9 @@ static void __exit vfio_cleanup(void)
 {
 	WARN_ON(!list_empty(&vfio.group_list));
 
-#ifdef CONFIG_VFIO_NOIOMMU
-	vfio_unregister_iommu_driver(&vfio_noiommu_ops);
-#endif
+	if (IS_ENABLED(CONFIG_VFIO_NOIOMMU))
+		vfio_unregister_iommu_driver(&vfio_noiommu_ops);
+
 	ida_destroy(&vfio.group_ida);
 	unregister_chrdev_region(vfio.group_devt, MINORMASK + 1);
 	class_destroy(vfio.class);
Signature · 1 line
2.37.2
#20 lore raw

RE: [PATCH 4/8] vfio: Remove #ifdefs around CONFIG_VFIO_NOIOMMU

Quoted 13 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM
#ifdef CONFIG_VFIO_NOIOMMU

drivers/vfio/vfio_main.c +2 -2
-static bool noiommu __read_mostly;
+static bool vfio_noiommu __read_mostly;
 module_param_named(enable_unsafe_noiommu_mode,
-		   noiommu, bool, S_IRUGO | S_IWUSR);
+		   vfio_noiommu, bool, S_IRUGO | S_IWUSR);
 MODULE_PARM_DESC(enable_unsafe_noiommu_mode, "Enable UNSAFE,

no-IOMMU mode. This mode provides no device isolation, no DMA translation, no host kernel protection, cannot be used for device assignment to virtual machines, requires RAWIO permissions, and will taint the kernel. If you do not know what this is for, step away. (default: false)");

drivers/vfio/vfio_main.c struct vfio_group { · line 91

Quoted 3 lines
drivers/vfio/vfio_main.c +2 -0
+#else
+enum { vfio_noiommu = false };
 #endif

what is the benefit of enum here?

#21 lore raw

Re: [PATCH 4/8] vfio: Remove #ifdefs around CONFIG_VFIO_NOIOMMU

Quoted 10 lines · Kevin
Quoted 10 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM
#ifdef CONFIG_VFIO_NOIOMMU

drivers/vfio/vfio_main.c +2 -2
-static bool noiommu __read_mostly;
+static bool vfio_noiommu __read_mostly;
 module_param_named(enable_unsafe_noiommu_mode,
-		   noiommu, bool, S_IRUGO | S_IWUSR);
+		   vfio_noiommu, bool, S_IRUGO | S_IWUSR);
 MODULE_PARM_DESC(enable_unsafe_noiommu_mode, "Enable UNSAFE,

no-IOMMU mode. This mode provides no device isolation, no DMA

Quoted 8 lines

drivers/vfio/vfio_main.c struct vfio_group { · line 91

Quoted 6 lines

translation, no host kernel protection, cannot be used for device assignment to virtual machines, requires RAWIO permissions, and will taint the kernel. If you do not know what this is for, step away. (default: false)");

drivers/vfio/vfio_main.c +2 -0
+#else
+enum { vfio_noiommu = false };
 #endif

what is the benefit of enum here?

It means we don't have to use #ifdef to protect references to vfio_noiommu. Do mean enum vs #define? I prefer generally prefer enums as they behave more like a variable.

Jason

Reviewed-by
#22 lore raw

RE: [PATCH 4/8] vfio: Remove #ifdefs around CONFIG_VFIO_NOIOMMU

Quoted 26 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 11:29 PM

Quoted 9 lines · Kevin

drivers/vfio/vfio_main.c struct vfio_group {

Quoted 9 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM
#ifdef CONFIG_VFIO_NOIOMMU

drivers/vfio/vfio_main.c +2 -2
-static bool noiommu __read_mostly;
+static bool vfio_noiommu __read_mostly;
 module_param_named(enable_unsafe_noiommu_mode,
-		   noiommu, bool, S_IRUGO | S_IWUSR);
+		   vfio_noiommu, bool, S_IRUGO | S_IWUSR);
 MODULE_PARM_DESC(enable_unsafe_noiommu_mode, "Enable

UNSAFE,

Quoted 2 lines
Quoted 2 lines

no-IOMMU mode. This mode provides no device isolation, no DMA translation, no host kernel protection, cannot be used for device

assignment

Quoted 1 line
Quoted 1 line

to virtual machines, requires RAWIO permissions, and will taint the kernel.

If

Quoted 6 lines

drivers/vfio/vfio_main.c struct vfio_group { · line 91

Quoted 4 lines

you do not know what this is for, step away. (default: false)");

drivers/vfio/vfio_main.c +2 -0
+#else
+enum { vfio_noiommu = false };
 #endif

what is the benefit of enum here?

Quoted 4 lines

It means we don't have to use #ifdef to protect references to vfio_noiommu. Do mean enum vs #define? I prefer generally prefer enums as they behave more like a variable.

Okay.

#23 lore raw

PATCH 5/8 vfio: Split out container code from the init/cleanup functions

In reply to Jason Gunthorpe

This miscdev, noiommu driver and a coule of globals are all container items. Move this init into its own functions.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 52 +++++++++++++++++++++++++++-------------
 1 file changed, 35 insertions(+), 17 deletions(-)
drivers/vfio/vfio_main.c +35 -17
index f76cea312028a6..bfe13b5c12fed7 100644
@@ -2067,14 +2067,11 @@ static struct miscdevice vfio_dev = {
 	.mode = S_IRUGO | S_IWUGO,
 };
 
-static int __init vfio_init(void)
+static int __init vfio_container_init(void)
 {
 	int ret;
 
-	ida_init(&vfio.group_ida);
-	mutex_init(&vfio.group_lock);
 	mutex_init(&vfio.iommu_drivers_lock);
-	INIT_LIST_HEAD(&vfio.group_list);
 	INIT_LIST_HEAD(&vfio.iommu_drivers_list);
 
 	ret = misc_register(&vfio_dev);
@@ -2083,6 +2080,38 @@ static int __init vfio_init(void)
 		return ret;
 	}
 
+	if (IS_ENABLED(CONFIG_VFIO_NOIOMMU)) {
+		ret = vfio_register_iommu_driver(&vfio_noiommu_ops);
+		if (ret)
+			goto err_misc;
+	}
+	return 0;
+
+err_misc:
+	misc_deregister(&vfio_dev);
+	return ret;
+}
+
+static void vfio_container_cleanup(void)
+{
+	if (IS_ENABLED(CONFIG_VFIO_NOIOMMU))
+		vfio_unregister_iommu_driver(&vfio_noiommu_ops);
+	misc_deregister(&vfio_dev);
+	mutex_destroy(&vfio.iommu_drivers_lock);
+}
+
+static int __init vfio_init(void)
+{
+	int ret;
+
+	ida_init(&vfio.group_ida);
+	mutex_init(&vfio.group_lock);
+	INIT_LIST_HEAD(&vfio.group_list);
+
+	ret = vfio_container_init();
+	if (ret)
+		return ret;
+
 	/* /dev/vfio/$GROUP */
 	vfio.class = class_create(THIS_MODULE, "vfio");
 	if (IS_ERR(vfio.class)) {
@@ -2096,22 +2125,14 @@ static int __init vfio_init(void)
 	if (ret)
 		goto err_alloc_chrdev;
 
-	if (IS_ENABLED(CONFIG_VFIO_NOIOMMU)) {
-		ret = vfio_register_iommu_driver(&vfio_noiommu_ops);
-		if (ret)
-			goto err_driver_register;
-	}
-
 	pr_info(DRIVER_DESC " version: " DRIVER_VERSION "\n");
 	return 0;
 
-err_driver_register:
-	unregister_chrdev_region(vfio.group_devt, MINORMASK + 1);
 err_alloc_chrdev:
 	class_destroy(vfio.class);
 	vfio.class = NULL;
 err_class:
-	misc_deregister(&vfio_dev);
+	vfio_container_cleanup();
 	return ret;
 }
 
@@ -2119,14 +2140,11 @@ static void __exit vfio_cleanup(void)
 {
 	WARN_ON(!list_empty(&vfio.group_list));
 
-	if (IS_ENABLED(CONFIG_VFIO_NOIOMMU))
-		vfio_unregister_iommu_driver(&vfio_noiommu_ops);
-
 	ida_destroy(&vfio.group_ida);
 	unregister_chrdev_region(vfio.group_devt, MINORMASK + 1);
 	class_destroy(vfio.class);
+	vfio_container_cleanup();
 	vfio.class = NULL;
-	misc_deregister(&vfio_dev);
 	xa_destroy(&vfio_device_set_xa);
 }
 
Signature · 1 line
2.37.2
Reviewed-by on v1 5/8
#24 lore raw

RE: [PATCH 5/8] vfio: Split out container code from the init/cleanup functions

Quoted 10 lines

From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, August 31, 2022 9:02 AM

This miscdev, noiommu driver and a coule of globals are all container items. Move this init into its own functions.

A following patch will move the vfio_container functions to their own .c file.

#25 lore raw

PATCH 6/8 vfio: Rename vfio_ioctl_check_extension()

In reply to Jason Gunthorpe

To vfio_container_ioctl_check_extension().

A following patch will turn this into a non-static function, make it clear it is related to the container.

---
 drivers/vfio/vfio_main.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
drivers/vfio/vfio_main.c +6 -5
index bfe13b5c12fed7..e1a424d243351a 100644
@@ -705,8 +705,9 @@ EXPORT_SYMBOL_GPL(vfio_unregister_group_dev);
 /*
  * VFIO base fd, /dev/vfio/vfio
  */
-static long vfio_ioctl_check_extension(struct vfio_container *container,
-				       unsigned long arg)
+static long
+vfio_container_ioctl_check_extension(struct vfio_container *container,
+				     unsigned long arg)
 {
 	struct vfio_iommu_driver *driver;
 	long ret = 0;
@@ -863,7 +864,7 @@ static long vfio_fops_unl_ioctl(struct file *filep,
 		ret = VFIO_API_VERSION;
 		break;
 	case VFIO_CHECK_EXTENSION:
-		ret = vfio_ioctl_check_extension(container, arg);
+		ret = vfio_container_ioctl_check_extension(container, arg);
 		break;
 	case VFIO_SET_IOMMU:
 		ret = vfio_ioctl_set_iommu(container, arg);
@@ -1770,8 +1771,8 @@ bool vfio_file_enforced_coherent(struct file *file)
 
 	down_read(&group->group_rwsem);
 	if (group->container) {
-		ret = vfio_ioctl_check_extension(group->container,
-						 VFIO_DMA_CC_IOMMU);
+		ret = vfio_container_ioctl_check_extension(group->container,
+							   VFIO_DMA_CC_IOMMU);
 	} else {
 		/*
 		 * Since the coherency state is determined only once a container
Signature · 1 line
2.37.2
#27 lore raw

PATCH 7/8 vfio: Split the register_device ops call into functions

In reply to Jason Gunthorpe

This is a container item.

A following patch will move the vfio_container functions to their own .c file.

---
 drivers/vfio/vfio_main.c | 39 +++++++++++++++++++++++----------------
 1 file changed, 23 insertions(+), 16 deletions(-)
drivers/vfio/vfio_main.c +23 -16
index e1a424d243351a..cf4b5418f497a9 100644
@@ -1110,9 +1110,28 @@ static void vfio_device_unassign_container(struct vfio_device *device)
 	up_write(&device->group->group_rwsem);
 }
 
+static void vfio_container_register_device(struct vfio_device *device)
+{
+	struct vfio_iommu_driver *iommu_driver =
+		device->group->container->iommu_driver;
+
+	if (iommu_driver && iommu_driver->ops->register_device)
+		iommu_driver->ops->register_device(
+			device->group->container->iommu_data, device);
+}
+
+static void vfio_container_unregister_device(struct vfio_device *device)
+{
+	struct vfio_iommu_driver *iommu_driver =
+		device->group->container->iommu_driver;
+
+	if (iommu_driver && iommu_driver->ops->unregister_device)
+		iommu_driver->ops->unregister_device(
+			device->group->container->iommu_data, device);
+}
+
 static struct file *vfio_device_open(struct vfio_device *device)
 {
-	struct vfio_iommu_driver *iommu_driver;
 	struct file *filep;
 	int ret;
 
@@ -1143,12 +1162,7 @@ static struct file *vfio_device_open(struct vfio_device *device)
 			if (ret)
 				goto err_undo_count;
 		}
-
-		iommu_driver = device->group->container->iommu_driver;
-		if (iommu_driver && iommu_driver->ops->register_device)
-			iommu_driver->ops->register_device(
-				device->group->container->iommu_data, device);
-
+		vfio_container_register_device(device);
 		up_read(&device->group->group_rwsem);
 	}
 	mutex_unlock(&device->dev_set->lock);
@@ -1186,10 +1200,7 @@ static struct file *vfio_device_open(struct vfio_device *device)
 	if (device->open_count == 1 && device->ops->close_device) {
 		device->ops->close_device(device);
 
-		iommu_driver = device->group->container->iommu_driver;
-		if (iommu_driver && iommu_driver->ops->unregister_device)
-			iommu_driver->ops->unregister_device(
-				device->group->container->iommu_data, device);
+		vfio_container_unregister_device(device);
 	}
 err_undo_count:
 	up_read(&device->group->group_rwsem);
@@ -1368,7 +1379,6 @@ static const struct file_operations vfio_group_fops = {
 static int vfio_device_fops_release(struct inode *inode, struct file *filep)
 {
 	struct vfio_device *device = filep->private_data;
-	struct vfio_iommu_driver *iommu_driver;
 
 	mutex_lock(&device->dev_set->lock);
 	vfio_assert_device_open(device);
@@ -1376,10 +1386,7 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
 	if (device->open_count == 1 && device->ops->close_device)
 		device->ops->close_device(device);
 
-	iommu_driver = device->group->container->iommu_driver;
-	if (iommu_driver && iommu_driver->ops->unregister_device)
-		iommu_driver->ops->unregister_device(
-			device->group->container->iommu_data, device);
+	vfio_container_unregister_device(device);
 	up_read(&device->group->group_rwsem);
 	device->open_count--;
 	if (device->open_count == 0)
Signature · 1 line
2.37.2
#29 lore raw

PATCH 8/8 vfio: Move container code into drivers/vfio/container.c

In reply to Jason Gunthorpe

All the functions that dereference struct vfio_container are moved into container.c.

Simple code motion, no functional change.

---
 drivers/vfio/Makefile    |   1 +
 drivers/vfio/container.c | 680 ++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio.h      |  47 +++
 drivers/vfio/vfio_main.c | 693 +--------------------------------------
 4 files changed, 729 insertions(+), 692 deletions(-)
 create mode 100644 drivers/vfio/container.c
drivers/vfio/Makefile +1 -0
index 1a32357592e3ea..d5ae6921eb4ece 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 vfio_virqfd-y := virqfd.o
 
+vfio-y += container.o
 vfio-y += vfio_main.o
 
 obj-$(CONFIG_VFIO) += vfio.o
drivers/vfio/container.c new file +680 -0

680 lines not on this page · raw message

drivers/vfio/vfio.h +47 -0

66 lines not on this page · raw message

drivers/vfio/vfio_main.c +1 -692

750 lines not on this page · raw message

Signature · 1 line
2.37.2