Skip to contentSkip to conversation

tee subsystem pin_user_pages for v5.8

GIT PULL Superseded Jens Wiklander 3 participants 13 messages May 26, 2020 – Aug 25, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello arm-soc maintainers,

Please pull this small patch converting the tee subsystem to use pin_user_pages() instead of get_user_pages().

Thanks,
Jens

The following changes since commit ae83d0b416db002fe95601e7f97f64b59514d936:

Linux 5.7-rc2 (2020-04-19 14:35:30 -0700)

are available in the Git repository at:

git://git.linaro.org:/people/jens.wiklander/linux-tee.git tags/tee-pin-user-pages-for-5.8

for you to fetch changes up to 37f6b4d5f47b600ec4ab6682c005a44a1bfca530:

tee: convert get_user_pages() --> pin_user_pages() (2020-05-26 10:42:41 +0200)

----------------------------------------------------------------
Converts tee subsystem to use pin_user_pages() instead of get_user_pages()

----------------------------------------------------------------
John Hubbard (1):

      tee: convert get_user_pages() --> pin_user_pages()
 drivers/tee/tee_shm.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)
#1 lore raw
Quoted 12 lines · Jens Wiklander

Hello arm-soc maintainers,

Please pull this small patch converting the tee subsystem to use pin_user_pages() instead of get_user_pages().

Thanks,
Jens

The following changes since commit ae83d0b416db002fe95601e7f97f64b59514d936:

Linux 5.7-rc2 (2020-04-19 14:35:30 -0700)

Quoted 7 lines

are available in the Git repository at:

git://git.linaro.org:/people/jens.wiklander/linux-tee.git tags/tee-pin-user-pages-for-5.8

for you to fetch changes up to 37f6b4d5f47b600ec4ab6682c005a44a1bfca530:

tee: convert get_user_pages() --> pin_user_pages() (2020-05-26 10:42:41 +0200)

Hi, I noticed this never got merged, but I don't see any follow-up here that retracts it. Is it still pending merge such that I should queue it for v5.10?

-Olof

#2 lore raw
Quoted 15 lines · Olof Johansson
Quoted 14 lines · Jens Wiklander

Hello arm-soc maintainers,

Please pull this small patch converting the tee subsystem to use pin_user_pages() instead of get_user_pages().

Thanks,
Jens

The following changes since commit ae83d0b416db002fe95601e7f97f64b59514d936:

   Linux 5.7-rc2 (2020-04-19 14:35:30 -0700)

are available in the Git repository at:

Quoted 9 lines
Quoted 5 lines

git://git.linaro.org:/people/jens.wiklander/linux-tee.git tags/tee-pin-user-pages-for-5.8

for you to fetch changes up to 37f6b4d5f47b600ec4ab6682c005a44a1bfca530:

   tee: convert get_user_pages() --> pin_user_pages() (2020-05-26 10:42:41 +0200)

Hi, I noticed this never got merged, but I don't see any follow-up here that retracts it. Is it still pending merge such that I should queue it for v5.10?

I think so. I had marked it in my notes as "accepted, and the maintainer will eventually merge it", and I left it at that. It's still desirable.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#3 lore raw
Quoted 21 lines · John Hubbard
Quoted 19 lines · Olof Johansson
Quoted 18 lines · Jens Wiklander

Hello arm-soc maintainers,

Please pull this small patch converting the tee subsystem to use pin_user_pages() instead of get_user_pages().

Thanks,
Jens

The following changes since commit ae83d0b416db002fe95601e7f97f64b59514d936:

   Linux 5.7-rc2 (2020-04-19 14:35:30 -0700)

are available in the Git repository at:

git://git.linaro.org:/people/jens.wiklander/linux-tee.git tags/tee-pin-user-pages-for-5.8

for you to fetch changes up to 37f6b4d5f47b600ec4ab6682c005a44a1bfca530:

Quoted 9 lines
Quoted 5 lines
Quoted 1 line
   tee: convert get_user_pages() --> pin_user_pages() (2020-05-26 10:42:41 +0200)

Hi, I noticed this never got merged, but I don't see any follow-up here that retracts it. Is it still pending merge such that I should queue it for v5.10?

I think so. I had marked it in my notes as "accepted, and the maintainer will eventually merge it", and I left it at that. It's still desirable.

Looks like it conflicts with some of the later work. Jens, given the timelines here it's probably easiest all around if you rebase/respin and send a fresh pull request. I could fix it up but you'd still need to review that so the amount of work is probably less if you do it directly.

-Olof

#4 lore raw
Quoted 33 lines · Olof Johansson
Quoted 21 lines · John Hubbard
Quoted 19 lines · Olof Johansson
Quoted 18 lines · Jens Wiklander

Hello arm-soc maintainers,

Please pull this small patch converting the tee subsystem to use pin_user_pages() instead of get_user_pages().

Thanks,
Jens

The following changes since commit ae83d0b416db002fe95601e7f97f64b59514d936:

   Linux 5.7-rc2 (2020-04-19 14:35:30 -0700)

are available in the Git repository at:

git://git.linaro.org:/people/jens.wiklander/linux-tee.git tags/tee-pin-user-pages-for-5.8

for you to fetch changes up to 37f6b4d5f47b600ec4ab6682c005a44a1bfca530:

Quoted 9 lines
Quoted 5 lines
Quoted 1 line
   tee: convert get_user_pages() --> pin_user_pages() (2020-05-26 10:42:41 +0200)

Hi, I noticed this never got merged, but I don't see any follow-up here that retracts it. Is it still pending merge such that I should queue it for v5.10?

I think so. I had marked it in my notes as "accepted, and the maintainer will eventually merge it", and I left it at that. It's still desirable.

Quoted 5 lines

Looks like it conflicts with some of the later work. Jens, given the timelines here it's probably easiest all around if you rebase/respin and send a fresh pull request. I could fix it up but you'd still need to review that so the amount of work is probably less if you do it directly.

Agree, I'll send a fresh pull request once we have this rebased. The conflict is with the recently added call to get_kernel_pages() when kernel memory is shared.
The conflict isn't trivial, I guess we need to handle the different types of pages differently when releasing them.
John, would you mind rebasing and posting the patch again?

Thanks,
Jens

#5 lore raw
Quoted 40 lines · Jens Wiklander
Quoted 33 lines · Olof Johansson
Quoted 21 lines · John Hubbard
Quoted 19 lines · Olof Johansson
Quoted 18 lines · Jens Wiklander

Hello arm-soc maintainers,

Please pull this small patch converting the tee subsystem to use pin_user_pages() instead of get_user_pages().

Thanks,
Jens

The following changes since commit ae83d0b416db002fe95601e7f97f64b59514d936:

    Linux 5.7-rc2 (2020-04-19 14:35:30 -0700)

are available in the Git repository at:

git://git.linaro.org:/people/jens.wiklander/linux-tee.git tags/tee-pin-user-pages-for-5.8

for you to fetch changes up to 37f6b4d5f47b600ec4ab6682c005a44a1bfca530:

Quoted 9 lines
Quoted 5 lines
Quoted 1 line
    tee: convert get_user_pages() --> pin_user_pages() (2020-05-26 10:42:41 +0200)

Hi, I noticed this never got merged, but I don't see any follow-up here that retracts it. Is it still pending merge such that I should queue it for v5.10?

I think so. I had marked it in my notes as "accepted, and the maintainer will eventually merge it", and I left it at that. It's still desirable.

Quoted 5 lines

Looks like it conflicts with some of the later work. Jens, given the timelines here it's probably easiest all around if you rebase/respin and send a fresh pull request. I could fix it up but you'd still need to review that so the amount of work is probably less if you do it directly.

Quoted 7 lines

Agree, I'll send a fresh pull request once we have this rebased. The conflict is with the recently added call to get_kernel_pages() when kernel memory is shared.
The conflict isn't trivial, I guess we need to handle the different types of pages differently when releasing them.
John, would you mind rebasing and posting the patch again?

Sure. Should it be against 5.9-rc2, or something else? I can do this in the morning, about 10 hrs from now.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#6 lore raw
Quoted 50 lines · John Hubbard
Quoted 40 lines · Jens Wiklander
Quoted 33 lines · Olof Johansson
Quoted 21 lines · John Hubbard
Quoted 19 lines · Olof Johansson
Quoted 18 lines · Jens Wiklander

Hello arm-soc maintainers,

Please pull this small patch converting the tee subsystem to use pin_user_pages() instead of get_user_pages().

Thanks,
Jens

The following changes since commit ae83d0b416db002fe95601e7f97f64b59514d936:

    Linux 5.7-rc2 (2020-04-19 14:35:30 -0700)

are available in the Git repository at:

git://git.linaro.org:/people/jens.wiklander/linux-tee.git tags/tee-pin-user-pages-for-5.8

for you to fetch changes up to 37f6b4d5f47b600ec4ab6682c005a44a1bfca530:

Quoted 9 lines
Quoted 5 lines
Quoted 1 line
    tee: convert get_user_pages() --> pin_user_pages() (2020-05-26 10:42:41 +0200)

Hi, I noticed this never got merged, but I don't see any follow-up here that retracts it. Is it still pending merge such that I should queue it for v5.10?

I think so. I had marked it in my notes as "accepted, and the maintainer will eventually merge it", and I left it at that. It's still desirable.

Quoted 5 lines

Looks like it conflicts with some of the later work. Jens, given the timelines here it's probably easiest all around if you rebase/respin and send a fresh pull request. I could fix it up but you'd still need to review that so the amount of work is probably less if you do it directly.

Quoted 7 lines

Agree, I'll send a fresh pull request once we have this rebased. The conflict is with the recently added call to get_kernel_pages() when kernel memory is shared.
The conflict isn't trivial, I guess we need to handle the different types of pages differently when releasing them.
John, would you mind rebasing and posting the patch again?

Quoted 2 lines

Sure. Should it be against 5.9-rc2, or something else? I can do this in the morning, about 10 hrs from now.

5.9-rc2 sounds good.

Thanks,
Jens

#7 lore raw

PATCH v2 1/1 tee: convert convert get_user_pages() --> pin_user_pages()

This code was using get_user_pages*(), in a "Case 2" scenario (DMA/RDMA), using the categorization from [1]. That means that it's time to convert the get_user_pages*() + put_page() calls to pin_user_pages*() + unpin_user_pages() calls.

There is some helpful background in [2]: basically, this is a small part of fixing a long-standing disconnect between pinning pages, and file systems' use of those pages.

[1] Documentation/core-api/pin_user_pages.rst

[2] "Explicit pinning of user-space pages":

    https://lwn.net/Articles/807108/

---

OK, this should be indentical to v1 [1], but now rebased against Linux 5.9-rc2.

As before, I've compile-tested it again with a cross compiler, but that's the only testing I'm set up for with CONFIG_TEE.

[1] https://lore.kernel.org/r/[email protected]

thanks,
John Hubbard
NVIDIA

 drivers/tee/tee_shm.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)
drivers/tee/tee_shm.c +3 -9
index 827ac3d0fea9..3c29e6c3ebe8 100644
@@ -32,16 +32,13 @@ static void tee_shm_release(struct tee_shm *shm)
 
 		poolm->ops->free(poolm, shm);
 	} else if (shm->flags & TEE_SHM_REGISTER) {
-		size_t n;
 		int rc = teedev->desc->ops->shm_unregister(shm->ctx, shm);
 
 		if (rc)
 			dev_err(teedev->dev.parent,
 				"unregister shm %p failed: %d", shm, rc);
 
-		for (n = 0; n < shm->num_pages; n++)
-			put_page(shm->pages[n]);
-
+		unpin_user_pages(shm->pages, shm->num_pages);
 		kfree(shm->pages);
 	}
 
@@ -228,7 +225,7 @@ struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr,
 	}
 
 	if (flags & TEE_SHM_USER_MAPPED) {
-		rc = get_user_pages_fast(start, num_pages, FOLL_WRITE,
+		rc = pin_user_pages_fast(start, num_pages, FOLL_WRITE,
 					 shm->pages);
 	} else {
 		struct kvec *kiov;
@@ -292,16 +289,13 @@ struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr,
 	return shm;
 err:
 	if (shm) {
-		size_t n;
-
 		if (shm->id >= 0) {
 			mutex_lock(&teedev->mutex);
 			idr_remove(&teedev->idr, shm->id);
 			mutex_unlock(&teedev->mutex);
 		}
 		if (shm->pages) {
-			for (n = 0; n < shm->num_pages; n++)
-				put_page(shm->pages[n]);
+			unpin_user_pages(shm->pages, shm->num_pages);
 			kfree(shm->pages);
 		}
 	}
Signature · 6 lines
2.28.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#8 lore raw

Re: [PATCH v2] tee: convert convert get_user_pages() --> pin_user_pages()

Quoted 21 lines · John Hubbard

This code was using get_user_pages*(), in a "Case 2" scenario (DMA/RDMA), using the categorization from [1]. That means that it's time to convert the get_user_pages*() + put_page() calls to pin_user_pages*() + unpin_user_pages() calls.

There is some helpful background in [2]: basically, this is a small part of fixing a long-standing disconnect between pinning pages, and file systems' use of those pages.

[1] Documentation/core-api/pin_user_pages.rst

[2] "Explicit pinning of user-space pages":

     https://lwn.net/Articles/807108/
Quoted 5 lines

---

OK, this should be indentical to v1 [1], but now rebased against Linux 5.9-rc2.

...ohhh, wait, I should have read the earlier message from Jens more carefully:

"The conflict isn't trivial, I guess we need to handle the different types of pages differently when releasing them."

So it's not good to have a logically identical patch. argghhh. Let me see how hard it is to track these memory types separately and handle the release accordingly, just a sec.

Sorry about the false move here.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
Quoted 52 lines

As before, I've compile-tested it again with a cross compiler, but that's the only testing I'm set up for with CONFIG_TEE.

[1] https://lore.kernel.org/r/[email protected]

thanks,
John Hubbard
NVIDIA

  drivers/tee/tee_shm.c | 12 +++---------
  1 file changed, 3 insertions(+), 9 deletions(-)
drivers/tee/tee_shm.c +2 -7
index 827ac3d0fea9..3c29e6c3ebe8 100644
@@ -32,16 +32,13 @@ static void tee_shm_release(struct tee_shm *shm)
 
  		poolm->ops->free(poolm, shm);
  	} else if (shm->flags & TEE_SHM_REGISTER) {
-		size_t n;
  		int rc = teedev->desc->ops->shm_unregister(shm->ctx, shm);
 
  		if (rc)
  			dev_err(teedev->dev.parent,
  				"unregister shm %p failed: %d", shm, rc);
 
-		for (n = 0; n < shm->num_pages; n++)
-			put_page(shm->pages[n]);
-
+		unpin_user_pages(shm->pages, shm->num_pages);
  		kfree(shm->pages);
  	}
 
@@ -228,7 +225,7 @@ struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr,
  	}
 
  	if (flags & TEE_SHM_USER_MAPPED) {
-		rc = get_user_pages_fast(start, num_pages, FOLL_WRITE,
+		rc = pin_user_pages_fast(start, num_pages, FOLL_WRITE,
  					 shm->pages);
  	} else {
  		struct kvec *kiov;
@@ -292,16 +289,13 @@ struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr,
  	return shm;
  err:
  	if (shm) {
-		size_t n;
-
  		if (shm->id >= 0) {
  			mutex_lock(&teedev->mutex);

drivers/tee/tee_shm.c struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr, · line 301

Quoted 10 lines
drivers/tee/tee_shm.c +1 -2
  			idr_remove(&teedev->idr, shm->id);
  			mutex_unlock(&teedev->mutex);
  		}
  		if (shm->pages) {
-			for (n = 0; n < shm->num_pages; n++)
-				put_page(shm->pages[n]);
+			unpin_user_pages(shm->pages, shm->num_pages);
  			kfree(shm->pages);
  		}
  	}

v

#9 lore raw

PATCH v3 1/1 tee: convert convert get_user_pages() --> pin_user_pages()

In reply to Jens Wiklander

This code was using get_user_pages*(), in a "Case 2" scenario (DMA/RDMA), using the categorization from [1]. That means that it's time to convert the get_user_pages*() + put_page() calls to pin_user_pages*() + unpin_user_pages() calls.

Factor out a new, small release_registered_pages() function, in order to consolidate the logic for discerning between TEE_SHM_USER_MAPPED and TEE_SHM_KERNEL_MAPPED pages. This also absorbs the kfree() call that is also required there.

There is some helpful background in [2]: basically, this is a small part of fixing a long-standing disconnect between pinning pages, and file systems' use of those pages.

[1] Documentation/core-api/pin_user_pages.rst

[2] "Explicit pinning of user-space pages":

    https://lwn.net/Articles/807108/

---

OK, one more try, this time actually handling the _USER_MAPPED vs. _KERNEL_MAPPED pages!

thanks,
John Hubbard
NVIDIA

 drivers/tee/tee_shm.c | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)
drivers/tee/tee_shm.c +19 -13
index 827ac3d0fea9..00472f5ce22e 100644
@@ -12,6 +12,22 @@
 #include <linux/uio.h>
 #include "tee_private.h"
 
+static void release_registered_pages(struct tee_shm *shm)
+{
+	if (shm->pages) {
+		if (shm->flags & TEE_SHM_USER_MAPPED) {
+			unpin_user_pages(shm->pages, shm->num_pages);
+		} else {
+			size_t n;
+
+			for (n = 0; n < shm->num_pages; n++)
+				put_page(shm->pages[n]);
+		}
+
+		kfree(shm->pages);
+	}
+}
+
 static void tee_shm_release(struct tee_shm *shm)
 {
 	struct tee_device *teedev = shm->ctx->teedev;
@@ -32,17 +48,13 @@ static void tee_shm_release(struct tee_shm *shm)
 
 		poolm->ops->free(poolm, shm);
 	} else if (shm->flags & TEE_SHM_REGISTER) {
-		size_t n;
 		int rc = teedev->desc->ops->shm_unregister(shm->ctx, shm);
 
 		if (rc)
 			dev_err(teedev->dev.parent,
 				"unregister shm %p failed: %d", shm, rc);
 
-		for (n = 0; n < shm->num_pages; n++)
-			put_page(shm->pages[n]);
-
-		kfree(shm->pages);
+		release_registered_pages(shm);
 	}
 
 	teedev_ctx_put(shm->ctx);
@@ -228,7 +240,7 @@ struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr,
 	}
 
 	if (flags & TEE_SHM_USER_MAPPED) {
-		rc = get_user_pages_fast(start, num_pages, FOLL_WRITE,
+		rc = pin_user_pages_fast(start, num_pages, FOLL_WRITE,
 					 shm->pages);
 	} else {
 		struct kvec *kiov;
@@ -292,18 +304,12 @@ struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr,
 	return shm;
 err:
 	if (shm) {
-		size_t n;
-
 		if (shm->id >= 0) {
 			mutex_lock(&teedev->mutex);
 			idr_remove(&teedev->idr, shm->id);
 			mutex_unlock(&teedev->mutex);
 		}
-		if (shm->pages) {
-			for (n = 0; n < shm->num_pages; n++)
-				put_page(shm->pages[n]);
-			kfree(shm->pages);
-		}
+		release_registered_pages(shm);
 	}
 	kfree(shm);
 	teedev_ctx_put(ctx);
Signature · 6 lines
2.28.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#10 lore raw

Re: [PATCH v3] tee: convert convert get_user_pages() --> pin_user_pages()

Quoted 26 lines · John Hubbard

This code was using get_user_pages*(), in a "Case 2" scenario (DMA/RDMA), using the categorization from [1]. That means that it's time to convert the get_user_pages*() + put_page() calls to pin_user_pages*() + unpin_user_pages() calls.

Factor out a new, small release_registered_pages() function, in order to consolidate the logic for discerning between TEE_SHM_USER_MAPPED and TEE_SHM_KERNEL_MAPPED pages. This also absorbs the kfree() call that is also required there.

There is some helpful background in [2]: basically, this is a small part of fixing a long-standing disconnect between pinning pages, and file systems' use of those pages.

[1] Documentation/core-api/pin_user_pages.rst

[2] "Explicit pinning of user-space pages":

    https://lwn.net/Articles/807108/
Quoted 8 lines

---

OK, one more try, this time actually handling the _USER_MAPPED vs. _KERNEL_MAPPED pages!

thanks,
John Hubbard
NVIDIA

Looks good and it works too! :-) I've tested it on my Hikey board with the OP-TEE test suite.
I'm picking this up.

Thanks,
Jens

#11 lore raw

Re: [PATCH v3] tee: convert convert get_user_pages() --> pin_user_pages()

Quoted 1 line · Jens Wiklander

On Mon, Aug 24, 2020 at 02:11:25PM -0700, John Hubbard wrote:

...

Quoted 11 lines
Quoted 6 lines

OK, one more try, this time actually handling the _USER_MAPPED vs. _KERNEL_MAPPED pages!

thanks,
John Hubbard
NVIDIA

Looks good and it works too! :-) I've tested it on my Hikey board with the OP-TEE test suite.
I'm picking this up.

Great! I see that I have, once again, somehow doubled up on the subject line:
"tee: convert convert ...". This particular typo just seems to stick to me. :)

If you get a chance to fix that up by changing it to just a single "convert" I'd appreciate it.

thanks,

Signature · 6 lines
John Hubbard
NVIDIA
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#12 lore raw

Re: [PATCH v3] tee: convert convert get_user_pages() --> pin_user_pages()

Quoted 16 lines · John Hubbard
Quoted 1 line · Jens Wiklander

On Mon, Aug 24, 2020 at 02:11:25PM -0700, John Hubbard wrote:

...

Quoted 11 lines
Quoted 6 lines

OK, one more try, this time actually handling the _USER_MAPPED vs. _KERNEL_MAPPED pages!

thanks,
John Hubbard
NVIDIA

Looks good and it works too! :-) I've tested it on my Hikey board with the OP-TEE test suite.
I'm picking this up.

Quoted 5 lines

Great! I see that I have, once again, somehow doubled up on the subject line:
"tee: convert convert ...". This particular typo just seems to stick to me. :)

If you get a chance to fix that up by changing it to just a single "convert" I'd appreciate it.

Sure, no problem.

Cheers,
Jens