Skip to contentSkip to conversation

fpga: improve protection against low-level control module unloading

RFC PATCH In review Marco Pagani 2 participants 17 messages Jan 11, 2024 – Mar 4, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v5 cover fpga: improve protection against low-level control module unloading

This RFC proposes a solution to keep protecting the fpga manager against the unloading of the low-level control modules while addressing the limitations of the current implementation. Currently, the code assumes that the low-level module registers a driver for the parent device that is later used to take the module's refcount. This proposal removes this limitation by adding a module owner field to the fpga_manager struct that can be set while registering the manager.

For more context, please refer to these threads:
https://lore.kernel.org/linux-fpga/ZS6hhlvjUcqyv8zL@yilunxu-OptiPlex-7050
https://lore.kernel.org/linux-fpga/ZT9qENE9fE3Z0KCW@yilunxu-OptiPlex-7050

v5:
- Updated the documentation
- Removed kernel-doc comments for helper macros v4:
- Use helper macros to set the owner module v3:
- Improved locking
v2:
- Fixed protection against races during module removal

Marco Pagani (1):
fpga: add an owner and use it to take the low-level module's refcount

 Documentation/driver-api/fpga/fpga-mgr.rst | 34 ++++----
 drivers/fpga/fpga-mgr.c                    | 93 ++++++++++++++--------
 include/linux/fpga/fpga-mgr.h              | 28 +++++--
 3 files changed, 102 insertions(+), 53 deletions(-)

base-commit: c849ecb2ae8413f86c84627cb0af06dffce4e215

Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH v5 1/1 fpga: add an owner and use it to take the low-level module's refcount

Add a module owner field to the fpga_manager struct to take the low-level control module refcount instead of assuming that the parent device has a driver and using its owner pointer. The owner is now passed as an additional argument at registration time. To this end, the functions for registration have been modified to take an additional owner parameter and renamed to avoid conflicts. The old function names are now used for helper macros that automatically set the module that registers the fpga manager as the owner. This ensures compatibility with existing low-level control modules and reduces the chances of registering a manager without setting the owner.

To detect when the owner module pointer becomes stale, set the mops pointer to null during fpga_mgr_unregister() and test it before taking the module's refcount. Use a mutex to protect against a crash that can happen if __fpga_mgr_get() gets suspended between testing the mops pointer and taking the refcount while the low-level module is being unloaded.

Update the documentation to keep it consistent with the new interface for registering an fpga manager.

Other changes: opportunistically move put_device() from __fpga_mgr_get() to fpga_mgr_get() and of_fpga_mgr_get() to improve code clarity since the device refcount in taken in these functions.

Fixes 654ba4cc0f3e ("fpga manager: ensure lifetime with of_fpga_mgr_get")
---
 Documentation/driver-api/fpga/fpga-mgr.rst | 34 ++++----
 drivers/fpga/fpga-mgr.c                    | 93 ++++++++++++++--------
 include/linux/fpga/fpga-mgr.h              | 28 +++++--
 3 files changed, 102 insertions(+), 53 deletions(-)
Documentation/driver-api/fpga/fpga-mgr.rst +20 -14
index 49c0a9512653..8d2b79f696c1 100644
@@ -24,7 +24,8 @@ How to support a new FPGA device
 --------------------------------
 
 To add another FPGA manager, write a driver that implements a set of ops.  The
-probe function calls fpga_mgr_register() or fpga_mgr_register_full(), such as::
+probe function calls ``fpga_mgr_register()`` or ``fpga_mgr_register_full()``,
+such as::
 
 	static const struct fpga_manager_ops socfpga_fpga_ops = {
 		.write_init = socfpga_fpga_ops_configure_init,
@@ -69,10 +70,11 @@ probe function calls fpga_mgr_register() or fpga_mgr_register_full(), such as::
 	}
 
 Alternatively, the probe function could call one of the resource managed
-register functions, devm_fpga_mgr_register() or devm_fpga_mgr_register_full().
-When these functions are used, the parameter syntax is the same, but the call
-to fpga_mgr_unregister() should be removed. In the above example, the
-socfpga_fpga_remove() function would not be required.
+register functions, ``devm_fpga_mgr_register()`` or
+``devm_fpga_mgr_register_full()``.  When these functions are used, the
+parameter syntax is the same, but the call to ``fpga_mgr_unregister()`` should be
+removed. In the above example, the ``socfpga_fpga_remove()`` function would not be
+required.
 
 The ops will implement whatever device specific register writes are needed to
 do the programming sequence for this particular FPGA.  These ops return 0 for
@@ -125,15 +127,19 @@ API for implementing a new FPGA Manager driver
 * struct fpga_manager -  the FPGA manager struct
 * struct fpga_manager_ops -  Low level FPGA manager driver ops
 * struct fpga_manager_info -  Parameter structure for fpga_mgr_register_full()
-* fpga_mgr_register_full() -  Create and register an FPGA manager using the
+* __fpga_mgr_register_full() -  Create and register an FPGA manager using the
   fpga_mgr_info structure to provide the full flexibility of options
-* fpga_mgr_register() -  Create and register an FPGA manager using standard
+* __fpga_mgr_register() -  Create and register an FPGA manager using standard
   arguments
-* devm_fpga_mgr_register_full() -  Resource managed version of
-  fpga_mgr_register_full()
-* devm_fpga_mgr_register() -  Resource managed version of fpga_mgr_register()
+* __devm_fpga_mgr_register_full() -  Resource managed version of
+  __fpga_mgr_register_full()
+* __devm_fpga_mgr_register() -  Resource managed version of __fpga_mgr_register()
 * fpga_mgr_unregister() -  Unregister an FPGA manager
 
+Helper macros ``fpga_mgr_register_full()``, ``fpga_mgr_register()``,
+``devm_fpga_mgr_register_full()``, and ``devm_fpga_mgr_register()`` are available
+to ease the registration.
+
 .. kernel-doc:: include/linux/fpga/fpga-mgr.h
    :functions: fpga_mgr_states
 
@@ -147,16 +153,16 @@ API for implementing a new FPGA Manager driver
    :functions: fpga_manager_info
 
 .. kernel-doc:: drivers/fpga/fpga-mgr.c
-   :functions: fpga_mgr_register_full
+   :functions: __fpga_mgr_register_full
 
 .. kernel-doc:: drivers/fpga/fpga-mgr.c
-   :functions: fpga_mgr_register
+   :functions: __fpga_mgr_register
 
 .. kernel-doc:: drivers/fpga/fpga-mgr.c
-   :functions: devm_fpga_mgr_register_full
+   :functions: __devm_fpga_mgr_register_full
 
 .. kernel-doc:: drivers/fpga/fpga-mgr.c
-   :functions: devm_fpga_mgr_register
+   :functions: __devm_fpga_mgr_register
 
 .. kernel-doc:: drivers/fpga/fpga-mgr.c
    :functions: fpga_mgr_unregister
drivers/fpga/fpga-mgr.c +60 -33

225 lines not on this page · raw message

include/linux/fpga/fpga-mgr.h +22 -6
index 54f63459efd6..0a4f67b59fb5 100644
@@ -201,6 +201,8 @@ struct fpga_manager_ops {
  * @state: state of fpga manager
  * @compat_id: FPGA manager id for compatibility check.
  * @mops: pointer to struct of fpga manager ops
+ * @mops_mutex: protects mops from low-level module removal
+ * @mops_owner: module containing the mops
  * @priv: low level driver private date
  */
 struct fpga_manager {
@@ -210,6 +212,8 @@ struct fpga_manager {
 	enum fpga_mgr_states state;
 	struct fpga_compat_id *compat_id;
 	const struct fpga_manager_ops *mops;
+	struct mutex mops_mutex;
+	struct module *mops_owner;
 	void *priv;
 };
 
@@ -230,18 +234,30 @@ struct fpga_manager *fpga_mgr_get(struct device *dev);
 
 void fpga_mgr_put(struct fpga_manager *mgr);
 
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
 
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);
+#define devm_fpga_mgr_register(parent, name, mops, priv) \
+	__devm_fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register(struct device *parent, const char *name,
-		       const struct fpga_manager_ops *mops, void *priv);
+__devm_fpga_mgr_register(struct device *parent, const char *name,
+			 const struct fpga_manager_ops *mops, void *priv,
+			 struct module *owner);
 
 #endif /*_LINUX_FPGA_MGR_H */
Signature · 1 line
2.43.0
#2 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 13 lines
include/linux/fpga/fpga-mgr.h +7 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,

include/linux/fpga/fpga-mgr.h struct fpga_manager *fpga_mgr_get(struct device *dev); · line 255

Quoted 10 lines
include/linux/fpga/fpga-mgr.h +6 -1
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
 
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

Next time if you think patches are ready for serious review and merge, drop the RFC prefix. That avoids an extra query.

Thanks,
Yilun

Quoted 14 lines
include/linux/fpga/fpga-mgr.h +5 -2
+#define devm_fpga_mgr_register(parent, name, mops, priv) \
+	__devm_fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register(struct device *parent, const char *name,
-		       const struct fpga_manager_ops *mops, void *priv);
+__devm_fpga_mgr_register(struct device *parent, const char *name,
+			 const struct fpga_manager_ops *mops, void *priv,
+			 struct module *owner);
 
 #endif /*_LINUX_FPGA_MGR_H */
Signature · 1 line
2.43.0
#3 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 17 lines · Xu Yilun
Quoted 17 lines
include/linux/fpga/fpga-mgr.h +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

include/linux/fpga/fpga-mgr.h struct fpga_manager *fpga_mgr_get(struct device *dev); · line 255

Quoted 6 lines
include/linux/fpga/fpga-mgr.h +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

Quoted 3 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

https://lore.kernel.org/linux-modules/[email protected]/

Quoted 2 lines

Next time if you think patches are ready for serious review and merge, drop the RFC prefix. That avoids an extra query.

Okay, I'll do it like that next time.

Thanks,
Marco

#4 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
include/linux/fpga/fpga-mgr.h +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

include/linux/fpga/fpga-mgr.h struct fpga_manager *fpga_mgr_get(struct device *dev); · line 255

Quoted 6 lines
include/linux/fpga/fpga-mgr.h +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 11 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

Thanks,
Yilun

Quoted 9 lines
Quoted 3 lines

Next time if you think patches are ready for serious review and merge, drop the RFC prefix. That avoids an extra query.

Okay, I'll do it like that next time.

Thanks,
Marco

#5 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
include/linux/fpga/fpga-mgr.h +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

include/linux/fpga/fpga-mgr.h struct fpga_manager *fpga_mgr_get(struct device *dev); · line 255

Quoted 6 lines
include/linux/fpga/fpga-mgr.h +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Thanks,
Marco

#6 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
include/linux/fpga/fpga-mgr.h +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

include/linux/fpga/fpga-mgr.h struct fpga_manager *fpga_mgr_get(struct device *dev); · line 255

Quoted 6 lines
include/linux/fpga/fpga-mgr.h +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

I have this concern when I was reviewing the same improvement for fpga bridge. The change for fpga bridge seems workable, the mutex keeps hold until fpga_bridge_put(). But I somewhat don't prefer the unregistration been unnecessarily blocked for long term.

I think your try_module_get_safe() patch may finally solve the invalid module owner issue. Some options now, we ignore the invalid module owner issue (it exists before this change) and merge the rest of the improvements, or we wait for your patch accepted then re-evaluate. I prefer the former.

Thanks,
Yilun

Quoted 4 lines

Thanks,
Marco

#7 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 79 lines · Xu Yilun
Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
include/linux/fpga/fpga-mgr.h +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

include/linux/fpga/fpga-mgr.h struct fpga_manager *fpga_mgr_get(struct device *dev); · line 255

Quoted 6 lines
include/linux/fpga/fpga-mgr.h +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}
Quoted 4 lines

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

How about moving mgr->mops = NULL from fpga_mgr_unregister() to class->dev_release()? In that way, mops will be set to NULL only when the manager dev refcount reaches 0.

If fpga_mgr_unregister() is called from module_exit(), we are sure that nobody got the manager dev earlier using fpga_mgr_get(), or it would have bumped up the module's refcount, preventing its removal in the first place. In this case, when device_unregister() is called, it will trigger dev_release() since the manager dev refcount has reached 0.

If fpga_mgr_unregister() is called elsewhere in the module that registered the manager (1), we have two subcases:

a) someone got the manager dev earlier and bumped the module's refcount. Hence, the ops are safe since the module cannot be removed until the manager dev is released by calling (the last) put_device(). This, in turn, will trigger class->dev_release().

b) no one got manager dev. In this case, class->dev_release() will be called immediately.

(1) The caller of fpga_mgr_register_*() is responsible for calling fpga_mgr_unregister(), as specified in the docs.

Quoted 4 lines

I have this concern when I was reviewing the same improvement for fpga bridge. The change for fpga bridge seems workable, the mutex keeps hold until fpga_bridge_put(). But I somewhat don't prefer the unregistration been unnecessarily blocked for long term.

I also don't like the idea of potentially blocking the unregistration, but I could not find a better solution for the bridge at the moment.

Quoted 5 lines

I think your try_module_get_safe() patch may finally solve the invalid module owner issue. Some options now, we ignore the invalid module owner issue (it exists before this change) and merge the rest of the improvements, or we wait for your patch accepted then re-evaluate. I prefer the former.

Yeah, try_module_get_safe() would make things simpler and easier. I'm currently working on a series of selftests to demonstrate that the function is safe from deadlocks, as requested by the maintainer. I hope I can convince people of the advantages.

Thanks,
Marco

#8 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 82 lines · Marco Pagani
Quoted 79 lines · Xu Yilun
Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
(quoted diff) +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}
Quoted 8 lines
Quoted 4 lines

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

How about moving mgr->mops = NULL from fpga_mgr_unregister() to class->dev_release()? In that way, mops will be set to NULL only when the manager dev refcount reaches 0.

I'm afraid it doesn't help. The lifecycle of the module and the fpga mgr dev is different.

We use mops = NULL to indicate module has been freed or will be freed in no time. On the other hand mops != NULL means module is still there, so that try_module_get() could be safely called. It is possible someone has got fpga mgr dev but not the module yet, at that time the module is unloaded, then try_module_get() triggers crash.

Quoted 3 lines

If fpga_mgr_unregister() is called from module_exit(), we are sure that nobody got the manager dev earlier using fpga_mgr_get(), or it would have bumped up

No, someone may get the manager dev but not the module yet, and been scheduled out.

Quoted 40 lines

the module's refcount, preventing its removal in the first place. In this case, when device_unregister() is called, it will trigger dev_release() since the manager dev refcount has reached 0.

If fpga_mgr_unregister() is called elsewhere in the module that registered the manager (1), we have two subcases:

a) someone got the manager dev earlier and bumped the module's refcount. Hence, the ops are safe since the module cannot be removed until the manager dev is released by calling (the last) put_device(). This, in turn, will trigger class->dev_release().

b) no one got manager dev. In this case, class->dev_release() will be called immediately.

(1) The caller of fpga_mgr_register_*() is responsible for calling fpga_mgr_unregister(), as specified in the docs.

Quoted 4 lines

I have this concern when I was reviewing the same improvement for fpga bridge. The change for fpga bridge seems workable, the mutex keeps hold until fpga_bridge_put(). But I somewhat don't prefer the unregistration been unnecessarily blocked for long term.

I also don't like the idea of potentially blocking the unregistration, but I could not find a better solution for the bridge at the moment.

Quoted 5 lines

I think your try_module_get_safe() patch may finally solve the invalid module owner issue. Some options now, we ignore the invalid module owner issue (it exists before this change) and merge the rest of the improvements, or we wait for your patch accepted then re-evaluate. I prefer the former.

Yeah, try_module_get_safe() would make things simpler and easier. I'm currently working on a series of selftests to demonstrate that the function is safe from deadlocks, as requested by the maintainer. I hope I can convince people of the advantages.

Thanks,
Marco

#9 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 102 lines · Xu Yilun
Quoted 82 lines · Marco Pagani
Quoted 79 lines · Xu Yilun
Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
(quoted diff) +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}
Quoted 8 lines
Quoted 4 lines

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

How about moving mgr->mops = NULL from fpga_mgr_unregister() to class->dev_release()? In that way, mops will be set to NULL only when the manager dev refcount reaches 0.

I'm afraid it doesn't help. The lifecycle of the module and the fpga mgr dev is different.

We use mops = NULL to indicate module has been freed or will be freed in no time. On the other hand mops != NULL means module is still there, so that try_module_get() could be safely called. It is possible someone has got fpga mgr dev but not the module yet, at that time the module is unloaded, then try_module_get() triggers crash.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

If fpga_mgr_unregister() is called from module_exit(), we are sure that nobody got the manager dev earlier using fpga_mgr_get(), or it would have bumped up

No, someone may get the manager dev but not the module yet, and been scheduled out.

You are right. Overall, it's a bad idea. How about then using an additional bool flag instead of "overloading" the mops pointer? Something like:

get:

	if (!mgr->owner_valid || !try_module_get(mgr->mops_owner))

remove:

	mgr->owner_valid = false;

Another possibility that comes to my mind would be to "overload" the owner pointer itself by using the ERR_PTR/IS_ERR macros. However, it looks ugly to me.

Thanks,
Marco

[...]

#10 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 112 lines · Marco Pagani
Quoted 102 lines · Xu Yilun
Quoted 82 lines · Marco Pagani
Quoted 79 lines · Xu Yilun
Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
(quoted diff) +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}
Quoted 8 lines
Quoted 4 lines

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

How about moving mgr->mops = NULL from fpga_mgr_unregister() to class->dev_release()? In that way, mops will be set to NULL only when the manager dev refcount reaches 0.

I'm afraid it doesn't help. The lifecycle of the module and the fpga mgr dev is different.

We use mops = NULL to indicate module has been freed or will be freed in no time. On the other hand mops != NULL means module is still there, so that try_module_get() could be safely called. It is possible someone has got fpga mgr dev but not the module yet, at that time the module is unloaded, then try_module_get() triggers crash.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

If fpga_mgr_unregister() is called from module_exit(), we are sure that nobody got the manager dev earlier using fpga_mgr_get(), or it would have bumped up

No, someone may get the manager dev but not the module yet, and been scheduled out.

Quoted 8 lines

You are right. Overall, it's a bad idea. How about then using an additional bool flag instead of "overloading" the mops pointer? Something like:

get:

	if (!mgr->owner_valid || !try_module_get(mgr->mops_owner))

remove:

	mgr->owner_valid = false;

I'm not quite sure which function is actually mentioned by "remove". I assume it should be fpga_mgr_unregister(). IIUC this flag means no more reference to fpga mgr, but existing references are still valid.

It works for me. But the name of this flag could be reconsidered to avoid misunderstanding. The owner is still valid (we still need to put the owner) but allows no more reference. Maybe "owner_inactive"?

I still wanna this owner reference change been splitted, so that we could simply revert it when the try_module_get_safe() got accepted.

Thanks,
Yilun

Quoted 11 lines

Another possibility that comes to my mind would be to "overload" the owner pointer itself by using the ERR_PTR/IS_ERR macros. However, it looks ugly to me.

Thanks,
Marco

[...]

#11 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 116 lines · Xu Yilun
Quoted 115 lines · Marco Pagani
Quoted 102 lines · Xu Yilun
Quoted 82 lines · Marco Pagani
Quoted 79 lines · Xu Yilun
Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
(quoted diff) +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}
Quoted 8 lines
Quoted 4 lines

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

How about moving mgr->mops = NULL from fpga_mgr_unregister() to class->dev_release()? In that way, mops will be set to NULL only when the manager dev refcount reaches 0.

I'm afraid it doesn't help. The lifecycle of the module and the fpga mgr dev is different.

We use mops = NULL to indicate module has been freed or will be freed in no time. On the other hand mops != NULL means module is still there, so that try_module_get() could be safely called. It is possible someone has got fpga mgr dev but not the module yet, at that time the module is unloaded, then try_module_get() triggers crash.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

If fpga_mgr_unregister() is called from module_exit(), we are sure that nobody got the manager dev earlier using fpga_mgr_get(), or it would have bumped up

No, someone may get the manager dev but not the module yet, and been scheduled out.

You are right. Overall, it's a bad idea. How about then using an additional bool flag instead of "overloading" the mops pointer? Something like:

Quoted 8 lines
Quoted 5 lines

get:

	if (!mgr->owner_valid || !try_module_get(mgr->mops_owner))

remove:

	mgr->owner_valid = false;

I'm not quite sure which function is actually mentioned by "remove". I assume it should be fpga_mgr_unregister().

Yes, I was referring to fpga_mgr_unregister().

Quoted 2 lines

IIUC this flag means no more reference to fpga mgr, but existing references are still valid.

Yes.

Quoted 4 lines

It works for me. But the name of this flag could be reconsidered to avoid misunderstanding. The owner is still valid (we still need to put the owner) but allows no more reference. Maybe "owner_inactive"?

Right, owner_valid might be misleading. How about removing any reference to the owner module and name the flag unreg?

__fpga_mgr_get:

	if (mgr->unreg || !try_module_get(mgr->mops_owner))
		mgr = ERR_PTR(-ENODEV);

fpga_mgr_unregister:
mgr->unreg = true;

Quoted 2 lines

I still wanna this owner reference change been splitted, so that we could simply revert it when the try_module_get_safe() got accepted.

I guess it may take some time to have try_module_get_safe() accepted. What do you prefer to do with the bridge and the region in the meantime?

Thanks,
Marco

#12 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 135 lines · Marco Pagani
Quoted 116 lines · Xu Yilun
Quoted 115 lines · Marco Pagani
Quoted 102 lines · Xu Yilun
Quoted 82 lines · Marco Pagani
Quoted 79 lines · Xu Yilun
Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
(quoted diff) +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}
Quoted 8 lines
Quoted 4 lines

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

How about moving mgr->mops = NULL from fpga_mgr_unregister() to class->dev_release()? In that way, mops will be set to NULL only when the manager dev refcount reaches 0.

I'm afraid it doesn't help. The lifecycle of the module and the fpga mgr dev is different.

We use mops = NULL to indicate module has been freed or will be freed in no time. On the other hand mops != NULL means module is still there, so that try_module_get() could be safely called. It is possible someone has got fpga mgr dev but not the module yet, at that time the module is unloaded, then try_module_get() triggers crash.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

If fpga_mgr_unregister() is called from module_exit(), we are sure that nobody got the manager dev earlier using fpga_mgr_get(), or it would have bumped up

No, someone may get the manager dev but not the module yet, and been scheduled out.

You are right. Overall, it's a bad idea. How about then using an additional bool flag instead of "overloading" the mops pointer? Something like:

Quoted 8 lines
Quoted 5 lines

get:

	if (!mgr->owner_valid || !try_module_get(mgr->mops_owner))

remove:

	mgr->owner_valid = false;

I'm not quite sure which function is actually mentioned by "remove". I assume it should be fpga_mgr_unregister().

Yes, I was referring to fpga_mgr_unregister().

Quoted 2 lines

IIUC this flag means no more reference to fpga mgr, but existing references are still valid.

Yes.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

It works for me. But the name of this flag could be reconsidered to avoid misunderstanding. The owner is still valid (we still need to put the owner) but allows no more reference. Maybe "owner_inactive"?

Right, owner_valid might be misleading. How about removing any reference to the owner module and name the flag unreg?

the full name "unregistered" is better.

Quoted 8 lines

__fpga_mgr_get:

	if (mgr->unreg || !try_module_get(mgr->mops_owner))
		mgr = ERR_PTR(-ENODEV);

fpga_mgr_unregister:
mgr->unreg = true;

Quoted 6 lines
Quoted 2 lines

I still wanna this owner reference change been splitted, so that we could simply revert it when the try_module_get_safe() got accepted.

I guess it may take some time to have try_module_get_safe() accepted. What do you prefer to do with the bridge and the region in the meantime?

This issue could happen in little chance. I actually don't have much preference, either way is good to me.

Thanks,
Yilun

Quoted 4 lines

Thanks,
Marco

#13 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 136 lines · Xu Yilun
Quoted 135 lines · Marco Pagani
Quoted 116 lines · Xu Yilun
Quoted 115 lines · Marco Pagani
Quoted 102 lines · Xu Yilun
Quoted 82 lines · Marco Pagani
Quoted 79 lines · Xu Yilun
Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
(quoted diff) +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}
Quoted 8 lines
Quoted 4 lines

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

How about moving mgr->mops = NULL from fpga_mgr_unregister() to class->dev_release()? In that way, mops will be set to NULL only when the manager dev refcount reaches 0.

I'm afraid it doesn't help. The lifecycle of the module and the fpga mgr dev is different.

We use mops = NULL to indicate module has been freed or will be freed in no time. On the other hand mops != NULL means module is still there, so that try_module_get() could be safely called. It is possible someone has got fpga mgr dev but not the module yet, at that time the module is unloaded, then try_module_get() triggers crash.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

If fpga_mgr_unregister() is called from module_exit(), we are sure that nobody got the manager dev earlier using fpga_mgr_get(), or it would have bumped up

No, someone may get the manager dev but not the module yet, and been scheduled out.

You are right. Overall, it's a bad idea. How about then using an additional bool flag instead of "overloading" the mops pointer? Something like:

Quoted 8 lines
Quoted 5 lines

get:

	if (!mgr->owner_valid || !try_module_get(mgr->mops_owner))

remove:

	mgr->owner_valid = false;

I'm not quite sure which function is actually mentioned by "remove". I assume it should be fpga_mgr_unregister().

Yes, I was referring to fpga_mgr_unregister().

Quoted 2 lines

IIUC this flag means no more reference to fpga mgr, but existing references are still valid.

Yes.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

It works for me. But the name of this flag could be reconsidered to avoid misunderstanding. The owner is still valid (we still need to put the owner) but allows no more reference. Maybe "owner_inactive"?

Right, owner_valid might be misleading. How about removing any reference to the owner module and name the flag unreg?

the full name "unregistered" is better.

That's fine by me.

Quoted 12 lines
Quoted 12 lines

__fpga_mgr_get:

	if (mgr->unreg || !try_module_get(mgr->mops_owner))
		mgr = ERR_PTR(-ENODEV);

fpga_mgr_unregister:
mgr->unreg = true;

Quoted 2 lines

I still wanna this owner reference change been splitted, so that we could simply revert it when the try_module_get_safe() got accepted.

Quoted 7 lines
Quoted 3 lines

I guess it may take some time to have try_module_get_safe() accepted. What do you prefer to do with the bridge and the region in the meantime?

This issue could happen in little chance. I actually don't have much preference, either way is good to me.

Okay, I'll also send the patch for the region then.

Thanks,
Marco

#14 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

In reply to Xu Yilun
Quoted 126 lines · Xu Yilun
Quoted 112 lines · Marco Pagani
Quoted 102 lines · Xu Yilun
Quoted 82 lines · Marco Pagani
Quoted 79 lines · Xu Yilun
Quoted 49 lines · Marco Pagani
Quoted 46 lines · Xu Yilun
Quoted 29 lines · Marco Pagani
Quoted 17 lines · Xu Yilun
Quoted 17 lines
(quoted diff) +9 -3
+#define fpga_mgr_register_full(parent, info) \
+	__fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			 struct module *owner);
 
+#define fpga_mgr_register(parent, name, mops, priv) \
+	__fpga_mgr_register(parent, name, mops, priv, THIS_MODULE)
 struct fpga_manager *
-fpga_mgr_register(struct device *parent, const char *name,
-		  const struct fpga_manager_ops *mops, void *priv);
+__fpga_mgr_register(struct device *parent, const char *name,
+		    const struct fpga_manager_ops *mops, void *priv, struct module *owner);
+
 void fpga_mgr_unregister(struct fpga_manager *mgr);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +4 -1
+#define devm_fpga_mgr_register_full(parent, info) \
+	__devm_fpga_mgr_register_full(parent, info, THIS_MODULE)
 struct fpga_manager *
-devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info);
+__devm_fpga_mgr_register_full(struct device *parent, const struct fpga_manager_info *info,
+			      struct module *owner);

Add a line here. I can do it myself if you agree.

Quoted 3 lines

Sure, that is fine by me. I also spotted a typo in the commit log body (in taken -> is taken). Do you want me to send a v6, or do you prefer to fix that in place?

No need, I can fix it.

Quoted 10 lines
Quoted 4 lines

There is still a RFC prefix for this patch. Are you ready to get it merged? If yes, Acked-by: Xu Yilun <[email protected]>

I'm ready for the patch to be merged. However, I recently sent an RFC to propose a safer implementation of try_module_get() that would simplify the code and may also benefit other subsystems. What do you think?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

https://lore.kernel.org/linux-modules/[email protected]/

I suggest take your fix to linux-fpga/for-next now. If your try_module_get() proposal is applied before the end of this cycle, we could re-evaluate this patch.

That's fine by me.

Sorry, I still found issues about this solution.

void fpga_mgr_unregister(struct fpga_manager *mgr)

{
        dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name);

        /*
         * If the low level driver provides a method for putting fpga into
         * a desired state upon unregister, do it.
         */
        fpga_mgr_fpga_remove(mgr);

        mutex_lock(&mgr->mops_mutex);

        mgr->mops = NULL;

        mutex_unlock(&mgr->mops_mutex);

        device_unregister(&mgr->dev);
}
Quoted 8 lines
Quoted 4 lines

Note that fpga_mgr_unregister() doesn't have to be called in module_exit(). So if we do fpga_mgr_get() then fpga_mgr_unregister(), We finally had a fpga_manager dev without mops, this is not what the user want and cause problem when using this fpga_manager dev for other FPGA APIs.

How about moving mgr->mops = NULL from fpga_mgr_unregister() to class->dev_release()? In that way, mops will be set to NULL only when the manager dev refcount reaches 0.

I'm afraid it doesn't help. The lifecycle of the module and the fpga mgr dev is different.

We use mops = NULL to indicate module has been freed or will be freed in no time. On the other hand mops != NULL means module is still there, so that try_module_get() could be safely called. It is possible someone has got fpga mgr dev but not the module yet, at that time the module is unloaded, then try_module_get() triggers crash.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

If fpga_mgr_unregister() is called from module_exit(), we are sure that nobody got the manager dev earlier using fpga_mgr_get(), or it would have bumped up

No, someone may get the manager dev but not the module yet, and been scheduled out.

Quoted 8 lines

You are right. Overall, it's a bad idea. How about then using an additional bool flag instead of "overloading" the mops pointer? Something like:

get:

	if (!mgr->owner_valid || !try_module_get(mgr->mops_owner))

remove:

	mgr->owner_valid = false;

I'm not quite sure which function is actually mentioned by "remove". I assume it should be fpga_mgr_unregister(). IIUC this flag means no more reference to fpga mgr, but existing references are still valid.

Quoted 7 lines

It works for me. But the name of this flag could be reconsidered to avoid misunderstanding. The owner is still valid (we still need to put the owner) but allows no more reference. Maybe "owner_inactive"?

I still wanna this owner reference change been splitted, so that we could simply revert it when the try_module_get_safe() got accepted.

Just to be sure that I understood correctly, you want to split the changes into two patches, like:

a) add module owner to the manager struct and take it in __fpga_mgr_get(); move put_device() from __fpga_mgr_get() to fpga_mgr_get() and of_fpga_mgr_get().

b) add the mutex and the unregistered flag for protection against races.

So that (b) can be reverted if try_module_get_safe() will be accepted?

Quoted 1 line

[...]

Thanks,
Marco

#15 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 10 lines

Just to be sure that I understood correctly, you want to split the changes into two patches, like:

a) add module owner to the manager struct and take it in __fpga_mgr_get(); move put_device() from __fpga_mgr_get() to fpga_mgr_get() and of_fpga_mgr_get().

b) add the mutex and the unregistered flag for protection against races.

So that (b) can be reverted if try_module_get_safe() will be accepted?

Yes, that's what I mean.

Quoted 7 lines
Quoted 2 lines

[...]

Thanks,
Marco

#16 lore raw

Re: [RFC PATCH v5 1/1] fpga: add an owner and use it to take the low-level module's refcount

Quoted 7 lines · Xu Yilun
Quoted 7 lines

Just to be sure that I understood correctly, you want to split the changes into two patches, like:

a) add module owner to the manager struct and take it in __fpga_mgr_get(); move put_device() from __fpga_mgr_get() to fpga_mgr_get() and of_fpga_mgr_get().

Quoted 6 lines
Quoted 3 lines

b) add the mutex and the unregistered flag for protection against races.

So that (b) can be reverted if try_module_get_safe() will be accepted?

Yes, that's what I mean.

On second thought, I would prefer to send only the patches (a) for the manager, bridge, and region for the moment. As you said, the chances of having a race are slim (no crash has ever been reported to date). So, after (a), I think it is worth focusing first on try_module_get_safe() since it could also be useful for other subsystems. The (b) patches could always be applied later if try_module_get_safe() is not accepted. Thanks,
Marco