Skip to contentSkip to conversation

binder_alloc: Move alloc_page() out of mmap_rwsem to reduce the lock duration

PATCH In review Barry Song 4 participants 11 messages Sep 2, 2024 – Sep 3, 2024

Subsystems (2)
lore
Share & tools
lore

Barry Song

author
#0 lore raw

PATCH 1/1 binder_alloc: Move alloc_page() out of mmap_rwsem to reduce the lock duration

From: Barry Song <[email protected]>

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.
A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.
On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

Cc Greg Kroah-Hartman <[email protected]>
Cc "Arve Hjønnevåg" <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Martijn Coenen <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Carlos Llamas <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
---
 drivers/android/binder_alloc.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/android/binder_alloc.c +14 -4
index b3acbc4174fb..f20074e23a7c 100644
@@ -227,13 +227,23 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 	if (!mmget_not_zero(alloc->mm))
 		return -ESRCH;
 
+	/*
+	 * Don't allocate page in mmap_write_lock, this can block
+	 * mmap_rwsem for a long time; Meanwhile, allocation failure
+	 * doesn't necessarily need to return -ENOMEM, if lru_page
+	 * has been installed, we can still return 0(success).
+	 */
+	page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
+
 	/*
 	 * Protected with mmap_sem in write mode as multiple tasks
 	 * might race to install the same page.
 	 */
 	mmap_write_lock(alloc->mm);
-	if (binder_get_installed_page(lru_page))
+	if (binder_get_installed_page(lru_page)) {
+		ret = 1;
 		goto out;
+	}
 
 	if (!alloc->vma) {
 		pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
@@ -241,7 +251,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 		goto out;
 	}
 
-	page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
 	if (!page) {
 		pr_err("%d: failed to allocate page\n", alloc->pid);
 		ret = -ENOMEM;
@@ -252,7 +261,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 	if (ret) {
 		pr_err("%d: %s failed to insert page at offset %lx with %d\n",
 		       alloc->pid, __func__, addr - alloc->buffer, ret);
-		__free_page(page);
 		ret = -ENOMEM;
 		goto out;
 	}
@@ -262,7 +270,9 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 out:
 	mmap_write_unlock(alloc->mm);
 	mmput_async(alloc->mm);
-	return ret;
+	if (ret && page)
+		__free_page(page);
+	return ret < 0 ? ret : 0;
 }
 
 static int binder_install_buffer_pages(struct binder_alloc *alloc,
Signature · 1 line
2.39.3 (Apple Git-146)
#1 lore raw
Quoted 40 lines · Barry Song

From: Barry Song <[email protected]>

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.
A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.
On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

Cc Greg Kroah-Hartman <[email protected]>
Cc "Arve Hjønnevåg" <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Martijn Coenen <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Carlos Llamas <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
---
 drivers/android/binder_alloc.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/android/binder_alloc.c +0 -0
index b3acbc4174fb..f20074e23a7c 100644

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 236

Quoted 10 lines
drivers/android/binder_alloc.c +7 -0
@@ -227,13 +227,23 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 	if (!mmget_not_zero(alloc->mm))
 		return -ESRCH;
 
+	/*
+	 * Don't allocate page in mmap_write_lock, this can block
+	 * mmap_rwsem for a long time; Meanwhile, allocation failure
+	 * doesn't necessarily need to return -ENOMEM, if lru_page
+	 * has been installed, we can still return 0(success).
+	 */
+	page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);

But now you are allocating new pages even if binder_get_installed_page() is an error, right? Doesn't that slow things down?

How was this benchmarked?

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 244

Quoted 9 lines
drivers/android/binder_alloc.c +3 -1
+
 	/*
 	 * Protected with mmap_sem in write mode as multiple tasks
 	 * might race to install the same page.
 	 */
 	mmap_write_lock(alloc->mm);
-	if (binder_get_installed_page(lru_page))
+	if (binder_get_installed_page(lru_page)) {
+		ret = 1;

That is not a valid error value :(

Quoted 22 lines
drivers/android/binder_alloc.c +1 -2
 		goto out;
+	}
 
 	if (!alloc->vma) {
 		pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
@@ -241,7 +251,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 		goto out;
 	}
 
-	page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
 	if (!page) {
 		pr_err("%d: failed to allocate page\n", alloc->pid);
 		ret = -ENOMEM;
@@ -252,7 +261,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 	if (ret) {
 		pr_err("%d: %s failed to insert page at offset %lx with %d\n",
 		       alloc->pid, __func__, addr - alloc->buffer, ret);
-		__free_page(page);
 		ret = -ENOMEM;
 		goto out;
 	}

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 275

Quoted 7 lines
drivers/android/binder_alloc.c +3 -1
@@ -262,7 +270,9 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 out:
 	mmap_write_unlock(alloc->mm);
 	mmput_async(alloc->mm);
-	return ret;
+	if (ret && page)
+		__free_page(page);
+	return ret < 0 ? ret : 0;

Please only use ? : for when you have to, otherwise please spell it out with a normal if statement:

	if (ret < 0)
		return ret;
	return 0;

But, this is abusing the fact that you set "ret = 1" above, which is going to trip someone up in the future as that is NOT a normal coding pattern we have in the kernel, sorry.

If you insist on this change, please rework it to not have that type of "positive means one thing, 0 means another, and negative means yet something else" please.

thanks,

greg k-h

author
#2 lore raw
Quoted 45 lines · Greg Kroah-Hartman
Quoted 43 lines · Barry Song

From: Barry Song <[email protected]>

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.
A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.
On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

Cc Greg Kroah-Hartman <[email protected]>
Cc "Arve Hjønnevåg" <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Martijn Coenen <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Carlos Llamas <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
---
 drivers/android/binder_alloc.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/android/binder_alloc.c +0 -0
index b3acbc4174fb..f20074e23a7c 100644
@@ -227,13 +227,23 @@ static int binder_install_single_page(struct binder_alloc *alloc,
      if (!mmget_not_zero(alloc->mm))
              return -ESRCH;
 
Quoted 10 lines

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 236

Quoted 7 lines
drivers/android/binder_alloc.c +7 -0
+     /*
+      * Don't allocate page in mmap_write_lock, this can block
+      * mmap_rwsem for a long time; Meanwhile, allocation failure
+      * doesn't necessarily need to return -ENOMEM, if lru_page
+      * has been installed, we can still return 0(success).
+      */
+     page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);

But now you are allocating new pages even if binder_get_installed_page() is an error, right? Doesn't that slow things down?

very very unlikely, as the ratio is only 1/2400 while write lock 100% blocks everyone.

Quoted 3 lines

How was this benchmarked?

i actually put Tangquan's profiling result:
"
However, according to Tangquan's 48-hour profiling
using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible."

Quoted 48 lines

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 244

Quoted 9 lines
drivers/android/binder_alloc.c +3 -1
+
      /*
       * Protected with mmap_sem in write mode as multiple tasks
       * might race to install the same page.
       */
      mmap_write_lock(alloc->mm);
-     if (binder_get_installed_page(lru_page))
+     if (binder_get_installed_page(lru_page)) {
+             ret = 1;

That is not a valid error value :(

Quoted 22 lines
drivers/android/binder_alloc.c +1 -2
              goto out;
+     }
 
      if (!alloc->vma) {
              pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
@@ -241,7 +251,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
              goto out;
      }
 
-     page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
      if (!page) {
              pr_err("%d: failed to allocate page\n", alloc->pid);
              ret = -ENOMEM;
@@ -252,7 +261,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
      if (ret) {
              pr_err("%d: %s failed to insert page at offset %lx with %d\n",
                     alloc->pid, __func__, addr - alloc->buffer, ret);
-             __free_page(page);
              ret = -ENOMEM;
              goto out;
      }

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 275

Quoted 7 lines
drivers/android/binder_alloc.c +3 -1
@@ -262,7 +270,9 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 out:
      mmap_write_unlock(alloc->mm);
      mmput_async(alloc->mm);
-     return ret;
+     if (ret && page)
+             __free_page(page);
+     return ret < 0 ? ret : 0;

Please only use ? : for when you have to, otherwise please spell it out with a normal if statement:

        if (ret < 0)
                return ret;
        return 0;
Quoted 7 lines

But, this is abusing the fact that you set "ret = 1" above, which is going to trip someone up in the future as that is NOT a normal coding pattern we have in the kernel, sorry.

If you insist on this change, please rework it to not have that type of "positive means one thing, 0 means another, and negative means yet something else" please.

I was trying to consolidate all free_page() calls into one place. Otherwise, we would need multiple free_page() calls. I'm perfectly fine with having more free_page() calls in both the ret = 1 and ret < 0 paths. In that case, the ret = 1
path can be removed if you prefer.

Quoted 4 lines

thanks,

greg k-h

Thanks
Barry

#3 lore raw
Quoted 58 lines · Barry Song
Quoted 45 lines · Greg Kroah-Hartman
Quoted 43 lines · Barry Song

From: Barry Song <[email protected]>

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.
A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.
On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

Cc Greg Kroah-Hartman <[email protected]>
Cc "Arve Hjønnevåg" <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Martijn Coenen <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Carlos Llamas <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
---
 drivers/android/binder_alloc.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/android/binder_alloc.c +0 -0
index b3acbc4174fb..f20074e23a7c 100644
@@ -227,13 +227,23 @@ static int binder_install_single_page(struct binder_alloc *alloc,
      if (!mmget_not_zero(alloc->mm))
              return -ESRCH;
 
Quoted 10 lines

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 236

Quoted 7 lines
drivers/android/binder_alloc.c +7 -0
+     /*
+      * Don't allocate page in mmap_write_lock, this can block
+      * mmap_rwsem for a long time; Meanwhile, allocation failure
+      * doesn't necessarily need to return -ENOMEM, if lru_page
+      * has been installed, we can still return 0(success).
+      */
+     page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);

But now you are allocating new pages even if binder_get_installed_page() is an error, right? Doesn't that slow things down?

Quoted 2 lines

very very unlikely, as the ratio is only 1/2400 while write lock 100% blocks everyone.

Ok, but:

Quoted 8 lines
Quoted 1 line

How was this benchmarked?

i actually put Tangquan's profiling result:
"
However, according to Tangquan's 48-hour profiling
using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible."

That's not a benchmark, or any real numbers of how this overall saves any time.

Quoted 55 lines
Quoted 47 lines

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 244

Quoted 8 lines
drivers/android/binder_alloc.c +2 -1
      /*
       * Protected with mmap_sem in write mode as multiple tasks
       * might race to install the same page.
       */
      mmap_write_lock(alloc->mm);
-     if (binder_get_installed_page(lru_page))
+     if (binder_get_installed_page(lru_page)) {
+             ret = 1;

That is not a valid error value :(

Quoted 22 lines
drivers/android/binder_alloc.c +1 -2
              goto out;
+     }
 
      if (!alloc->vma) {
              pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
@@ -241,7 +251,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
              goto out;
      }
 
-     page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
      if (!page) {
              pr_err("%d: failed to allocate page\n", alloc->pid);
              ret = -ENOMEM;
@@ -252,7 +261,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
      if (ret) {
              pr_err("%d: %s failed to insert page at offset %lx with %d\n",
                     alloc->pid, __func__, addr - alloc->buffer, ret);
-             __free_page(page);
              ret = -ENOMEM;
              goto out;
      }

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 275

Quoted 7 lines
drivers/android/binder_alloc.c +3 -1
@@ -262,7 +270,9 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 out:
      mmap_write_unlock(alloc->mm);
      mmput_async(alloc->mm);
-     return ret;
+     if (ret && page)
+             __free_page(page);
+     return ret < 0 ? ret : 0;

Please only use ? : for when you have to, otherwise please spell it out with a normal if statement:

        if (ret < 0)
                return ret;
        return 0;
Quoted 7 lines

But, this is abusing the fact that you set "ret = 1" above, which is going to trip someone up in the future as that is NOT a normal coding pattern we have in the kernel, sorry.

If you insist on this change, please rework it to not have that type of "positive means one thing, 0 means another, and negative means yet something else" please.

Quoted 5 lines

I was trying to consolidate all free_page() calls into one place. Otherwise, we would need multiple free_page() calls. I'm perfectly fine with having more free_page() calls in both the ret = 1 and ret < 0 paths. In that case, the ret = 1
path can be removed if you prefer.

Remember, we write code for people first, and compilers second. You have to maintain this code for the next 10+ years, make it _VERY_ obvious what is happening and how it works as you will be coming back to it and not remembering what was made for what reason at all.

thanks,

greg k-h

author
#4 lore raw
Quoted 74 lines · Greg Kroah-Hartman
Quoted 58 lines · Barry Song
Quoted 45 lines · Greg Kroah-Hartman
Quoted 43 lines · Barry Song

From: Barry Song <[email protected]>

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.
A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.
On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

Cc Greg Kroah-Hartman <[email protected]>
Cc "Arve Hjønnevåg" <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Martijn Coenen <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Carlos Llamas <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
---
 drivers/android/binder_alloc.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/android/binder_alloc.c +0 -0
index b3acbc4174fb..f20074e23a7c 100644
@@ -227,13 +227,23 @@ static int binder_install_single_page(struct binder_alloc *alloc,
      if (!mmget_not_zero(alloc->mm))
              return -ESRCH;
 
Quoted 10 lines

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 236

Quoted 7 lines
drivers/android/binder_alloc.c +7 -0
+     /*
+      * Don't allocate page in mmap_write_lock, this can block
+      * mmap_rwsem for a long time; Meanwhile, allocation failure
+      * doesn't necessarily need to return -ENOMEM, if lru_page
+      * has been installed, we can still return 0(success).
+      */
+     page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);

But now you are allocating new pages even if binder_get_installed_page() is an error, right? Doesn't that slow things down?

Quoted 2 lines

very very unlikely, as the ratio is only 1/2400 while write lock 100% blocks everyone.

Ok, but:

Quoted 8 lines
Quoted 1 line

How was this benchmarked?

i actually put Tangquan's profiling result:
"
However, according to Tangquan's 48-hour profiling
using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible."

Quoted 2 lines

That's not a benchmark, or any real numbers of how this overall saves any time.

My point is that mmap_rw is one of the most notorious locks impacting system performance. Our profiling indicates that this binder's rwsem lock is among the top two write locks, blocking page faults and other accesses to the VMA.

From the memory management perspective, I believe the improvement will be quite evident. However, I agree that we should include more data to show how reducing lock contention can enhance overall user experience and minimize UI lag.

Quoted 62 lines
Quoted 56 lines
Quoted 48 lines

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 244

Quoted 9 lines
drivers/android/binder_alloc.c +2 -1
      /*
       * Protected with mmap_sem in write mode as multiple tasks
       * might race to install the same page.
       */
      mmap_write_lock(alloc->mm);
-     if (binder_get_installed_page(lru_page))
+     if (binder_get_installed_page(lru_page)) {
+             ret = 1;

That is not a valid error value :(

Quoted 22 lines
drivers/android/binder_alloc.c +1 -2
              goto out;
+     }
 
      if (!alloc->vma) {
              pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
@@ -241,7 +251,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
              goto out;
      }
 
-     page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
      if (!page) {
              pr_err("%d: failed to allocate page\n", alloc->pid);
              ret = -ENOMEM;
@@ -252,7 +261,6 @@ static int binder_install_single_page(struct binder_alloc *alloc,
      if (ret) {
              pr_err("%d: %s failed to insert page at offset %lx with %d\n",
                     alloc->pid, __func__, addr - alloc->buffer, ret);
-             __free_page(page);
              ret = -ENOMEM;
              goto out;
      }

drivers/android/binder_alloc.c static int binder_install_single_page(struct binder_alloc *alloc, · line 275

Quoted 7 lines
drivers/android/binder_alloc.c +3 -1
@@ -262,7 +270,9 @@ static int binder_install_single_page(struct binder_alloc *alloc,
 out:
      mmap_write_unlock(alloc->mm);
      mmput_async(alloc->mm);
-     return ret;
+     if (ret && page)
+             __free_page(page);
+     return ret < 0 ? ret : 0;

Please only use ? : for when you have to, otherwise please spell it out with a normal if statement:

        if (ret < 0)
                return ret;
        return 0;
Quoted 7 lines

But, this is abusing the fact that you set "ret = 1" above, which is going to trip someone up in the future as that is NOT a normal coding pattern we have in the kernel, sorry.

If you insist on this change, please rework it to not have that type of "positive means one thing, 0 means another, and negative means yet something else" please.

Quoted 5 lines

I was trying to consolidate all free_page() calls into one place. Otherwise, we would need multiple free_page() calls. I'm perfectly fine with having more free_page() calls in both the ret = 1 and ret < 0 paths. In that case, the ret = 1
path can be removed if you prefer.

Quoted 4 lines

Remember, we write code for people first, and compilers second. You have to maintain this code for the next 10+ years, make it _VERY_ obvious what is happening and how it works as you will be coming back to it and not remembering what was made for what reason at all.

no objection to this.

Quoted 4 lines

thanks,

greg k-h

Thanks
Barry

#5 lore raw
In reply to Barry Song
Quoted 10 lines · Barry Song

From: Barry Song <[email protected]>

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.

I suspect concurrent allocators make things better wrt response, cutting alloc latency down to 10ms for instance in your scenario. Feel free to show figures given Tangquan's 48-hour profiling.

Quoted 10 lines

A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.
On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

If spin, better not before taking a look at vm_insert_page().

author
#6 lore raw
Quoted 13 lines · Hillf Danton
Quoted 10 lines · Barry Song

From: Barry Song <[email protected]>

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.

Quoted 3 lines

I suspect concurrent allocators make things better wrt response, cutting alloc latency down to 10ms for instance in your scenario. Feel free to show figures given Tangquan's 48-hour profiling.

Likely.

Concurrent allocators are quite common in PFs which occur in the same PTE. whoever gets PTL sets PTE, others free the allocated pages.

Quoted 12 lines
Quoted 11 lines

A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.
On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

Quoted 1 line

If spin, better not before taking a look at vm_insert_page().

I have patch 2/3 transitioning to mmap_read_lock, and per_vma_lock is currently in the
testing queue. At the moment, alloc->spin is in place, but I'm not entirely convinced
it's the best replacement for the write lock. Let's wait for Tangquan's test results.

Patch 2 is detailed below, but it has only passed the build-test phase so far, so
its result is uncertain. I'm sharing it early in case you find it interesting. And I
am not convinced Commit d1d8875c8c13 ("binder: fix UAF of alloc->vma in race with munmap()") is a correct fix to really avoid all UAF of alloc->vma.

[PATCH] binder_alloc: Don't use mmap_write_lock for installing page

Commit d1d8875c8c13 ("binder: fix UAF of alloc->vma in race with munmap()") uses the mmap_rwsem write lock to protect against a race condition with munmap, where the vma is detached by the write lock, but pages are zapped by the read lock. This approach is extremely expensive for the system, though perhaps less so for binder itself, as the write lock can block all other operations.

As an alternative, we could hold only the read lock and re-check that the vma hasn't been detached. To protect simultaneous page installation, we could use alloc->lock instead.

---
 drivers/android/binder_alloc.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)
drivers/android/binder_alloc.c +0 -0
index f20074e23a7c..a2281dfacbbc 100644
@@ -228,24 +228,17 @@ static int binder_install_single_page(struct

binder_alloc *alloc,

                return -ESRCH;

        /*
-        * Don't allocate page in mmap_write_lock, this can block
-        * mmap_rwsem for a long time; Meanwhile, allocation failure
-        * doesn't necessarily need to return -ENOMEM, if lru_page
-        * has been installed, we can still return 0(success).
+        * Allocation failure doesn't necessarily need to return -ENOMEM,
+        * if lru_page has been installed, we can still return 0(success).
+        * So, defer the !page check until after binder_get_installed_page()
+        * is completed.
         */
        page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);

- /*

-        * Protected with mmap_sem in write mode as multiple tasks
-        * might race to install the same page.
-        */

- mmap_write_lock(alloc->mm);
- if (binder_get_installed_page(lru_page)) {
- ret = 1;
- goto out;
- }
+ mmap_read_lock(alloc->mm);

-       if (!alloc->vma) {
+       /* vma might have been dropped or deattached */
+       if (!alloc->vma || !find_vma(alloc->mm, addr)) {
                pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
                ret = -ESRCH;
                goto out;
@@ -257,18 +250,27 @@ static int binder_install_single_page(struct
binder_alloc *alloc,
                goto out;
        }

+ spin_lock(&alloc->lock);
+ if (binder_get_installed_page(lru_page)) {

+               spin_unlock(&alloc->lock);
+               ret = 1;
+               goto out;
+       }
+
        ret = vm_insert_page(alloc->vma, addr, page);
        if (ret) {
                pr_err("%d: %s failed to insert page at offset %lx with %d\n",
                       alloc->pid, __func__, addr - alloc->buffer, ret);
+               spin_unlock(&alloc->lock);
                ret = -ENOMEM;
                goto out;
        }

        /* Mark page installation complete and safe to use */
        binder_set_installed_page(lru_page, page);
+       spin_unlock(&alloc->lock);
 out:
-       mmap_write_unlock(alloc->mm);
+       mmap_read_unlock(alloc->mm);
        mmput_async(alloc->mm);
        if (ret && page)
                __free_page(page);

--
2.39.3 (Apple Git-146)

maintainer
#7 lore raw
Quoted 5 lines · Barry Song
Quoted 4 lines · Hillf Danton
Quoted 2 lines · Barry Song

From: Barry Song <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.

Thanks for you patch Barry. So, we are aware of this contention and I've been working on a fix for it. See more about this below.

Quoted 5 lines
Quoted 4 lines

I suspect concurrent allocators make things better wrt response, cutting alloc latency down to 10ms for instance in your scenario. Feel free to show figures given Tangquan's 48-hour profiling.

Quoted 13 lines

Likely.

Concurrent allocators are quite common in PFs which occur in the same PTE. whoever gets PTL sets PTE, others free the allocated pages.

Quoted 7 lines
Quoted 7 lines

A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.

This is not negligible. In fact, it is the exact reason for the page allocation to be done with the mmap sem. If the first thread sleeps on vm_insert_page(), then binder gets into a bad state of multiple threads trying to reclaim pages that won't really be used. Memory pressure goes from bad to worst pretty quick.

FWIW, I believe this was first talked about here:
https://lore.kernel.org/all/[email protected]/

Quoted 77 lines
Quoted 6 lines
Quoted 4 lines

On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

If spin, better not before taking a look at vm_insert_page().

I have patch 2/3 transitioning to mmap_read_lock, and per_vma_lock is currently in the
testing queue. At the moment, alloc->spin is in place, but I'm not entirely convinced
it's the best replacement for the write lock. Let's wait for Tangquan's test results.

Patch 2 is detailed below, but it has only passed the build-test phase so far, so
its result is uncertain. I'm sharing it early in case you find it interesting. And I
am not convinced Commit d1d8875c8c13 ("binder: fix UAF of alloc->vma in race with munmap()") is a correct fix to really avoid all UAF of alloc->vma.

[PATCH] binder_alloc: Don't use mmap_write_lock for installing page

Commit d1d8875c8c13 ("binder: fix UAF of alloc->vma in race with munmap()") uses the mmap_rwsem write lock to protect against a race condition with munmap, where the vma is detached by the write lock, but pages are zapped by the read lock. This approach is extremely expensive for the system, though perhaps less so for binder itself, as the write lock can block all other operations.

As an alternative, we could hold only the read lock and re-check that the vma hasn't been detached. To protect simultaneous page installation, we could use alloc->lock instead.

---
 drivers/android/binder_alloc.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)
drivers/android/binder_alloc.c +7 -14
index f20074e23a7c..a2281dfacbbc 100644
@@ -228,24 +228,17 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
                return -ESRCH;
 
        /*
-        * Don't allocate page in mmap_write_lock, this can block
-        * mmap_rwsem for a long time; Meanwhile, allocation failure
-        * doesn't necessarily need to return -ENOMEM, if lru_page
-        * has been installed, we can still return 0(success).
+        * Allocation failure doesn't necessarily need to return -ENOMEM,
+        * if lru_page has been installed, we can still return 0(success).
+        * So, defer the !page check until after binder_get_installed_page()
+        * is completed.
         */
        page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
 
-       /*
-        * Protected with mmap_sem in write mode as multiple tasks
-        * might race to install the same page.
-        */
-       mmap_write_lock(alloc->mm);
-       if (binder_get_installed_page(lru_page)) {
-               ret = 1;
-               goto out;
-       }
+       mmap_read_lock(alloc->mm);
 
-       if (!alloc->vma) {
+       /* vma might have been dropped or deattached */
+       if (!alloc->vma || !find_vma(alloc->mm, addr)) {
                pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
                ret = -ESRCH;
                goto out;

drivers/android/binder_alloc.c static int binder_install_single_page(struct · line 254

Quoted 5 lines
drivers/android/binder_alloc.c +1 -0
@@ -257,18 +250,27 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
                goto out;
        }
 
+       spin_lock(&alloc->lock);

You can't hold a spinlock and then call vm_insert_page().

Quoted 24 lines
drivers/android/binder_alloc.c +9 -1
+       if (binder_get_installed_page(lru_page)) {
+               spin_unlock(&alloc->lock);
+               ret = 1;
+               goto out;
+       }
+
        ret = vm_insert_page(alloc->vma, addr, page);
        if (ret) {
                pr_err("%d: %s failed to insert page at offset %lx with %d\n",
                       alloc->pid, __func__, addr - alloc->buffer, ret);
+               spin_unlock(&alloc->lock);
                ret = -ENOMEM;
                goto out;
        }
 
        /* Mark page installation complete and safe to use */
        binder_set_installed_page(lru_page, page);
+       spin_unlock(&alloc->lock);
 out:
-       mmap_write_unlock(alloc->mm);
+       mmap_read_unlock(alloc->mm);
        mmput_async(alloc->mm);
        if (ret && page)
                __free_page(page);
Quoted 2 lines

--
2.39.3 (Apple Git-146)

Sorry, but as I mentioned, I've been working on fixing this contention by supporting concurrent "faults" in binder_install_single_page(). This is the appropriate fix. I should be sending a patch soon after working out the conflicts with the shrinker's callback.

Thanks,
--
Carlos Llamas

author
#8 lore raw
Quoted 16 lines · Carlos Llamas
Quoted 5 lines · Barry Song
Quoted 4 lines · Hillf Danton
Quoted 2 lines · Barry Song

From: Barry Song <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The mmap_write_lock() can block all access to the VMAs, for example page faults. Performing memory allocation while holding this lock may trigger direct reclamation, leading to others being queued in the rwsem for an extended period.
We've observed that the allocation can sometimes take more than 300ms, significantly blocking other threads. The user interface sometimes becomes less responsive as a result. To prevent this, let's move the allocation outside of the write lock.

Quoted 2 lines

Thanks for you patch Barry. So, we are aware of this contention and I've been working on a fix for it. See more about this below.

Cool, Carlos!

Quoted 20 lines
Quoted 6 lines
Quoted 5 lines

I suspect concurrent allocators make things better wrt response, cutting alloc latency down to 10ms for instance in your scenario. Feel free to show figures given Tangquan's 48-hour profiling.

Quoted 13 lines

Likely.

Concurrent allocators are quite common in PFs which occur in the same PTE. whoever gets PTL sets PTE, others free the allocated pages.

Quoted 7 lines
Quoted 7 lines

A potential side effect could be an extra alloc_page() for the second thread executing binder_install_single_page() while the first thread has done it earlier. However, according to Tangquan's 48-hour profiling using monkey, the likelihood of this occurring is minimal, with a ratio of only 1 in 2400. Compared to the significantly costly rwsem, this is negligible.

Quoted 8 lines

This is not negligible. In fact, it is the exact reason for the page allocation to be done with the mmap sem. If the first thread sleeps on vm_insert_page(), then binder gets into a bad state of multiple threads trying to reclaim pages that won't really be used. Memory pressure goes from bad to worst pretty quick.

FWIW, I believe this was first talked about here:
https://lore.kernel.org/all/[email protected]/

However, I'm not entirely convinced that this is a problem :-) Concurrent allocations like this can occur in many places, especially in PFs. Reclamation is not useless because it helps free up memory for others; it's not without value.
I also don't believe binder is one of the largest users executing concurrent allocations.

Quoted 83 lines
Quoted 83 lines
Quoted 8 lines
Quoted 6 lines

On the other hand, holding a write lock without making any VMA modifications appears questionable and likely incorrect. While this patch focuses on reducing the lock duration, future updates may aim to eliminate the write lock entirely.

If spin, better not before taking a look at vm_insert_page().

I have patch 2/3 transitioning to mmap_read_lock, and per_vma_lock is currently in the
testing queue. At the moment, alloc->spin is in place, but I'm not entirely convinced
it's the best replacement for the write lock. Let's wait for Tangquan's test results.

Patch 2 is detailed below, but it has only passed the build-test phase so far, so
its result is uncertain. I'm sharing it early in case you find it interesting. And I
am not convinced Commit d1d8875c8c13 ("binder: fix UAF of alloc->vma in race with munmap()") is a correct fix to really avoid all UAF of alloc->vma.

[PATCH] binder_alloc: Don't use mmap_write_lock for installing page

Commit d1d8875c8c13 ("binder: fix UAF of alloc->vma in race with munmap()") uses the mmap_rwsem write lock to protect against a race condition with munmap, where the vma is detached by the write lock, but pages are zapped by the read lock. This approach is extremely expensive for the system, though perhaps less so for binder itself, as the write lock can block all other operations.

As an alternative, we could hold only the read lock and re-check that the vma hasn't been detached. To protect simultaneous page installation, we could use alloc->lock instead.

---
 drivers/android/binder_alloc.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)
drivers/android/binder_alloc.c +7 -14
index f20074e23a7c..a2281dfacbbc 100644
@@ -228,24 +228,17 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
                return -ESRCH;
 
        /*
-        * Don't allocate page in mmap_write_lock, this can block
-        * mmap_rwsem for a long time; Meanwhile, allocation failure
-        * doesn't necessarily need to return -ENOMEM, if lru_page
-        * has been installed, we can still return 0(success).
+        * Allocation failure doesn't necessarily need to return -ENOMEM,
+        * if lru_page has been installed, we can still return 0(success).
+        * So, defer the !page check until after binder_get_installed_page()
+        * is completed.
         */
        page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
 
-       /*
-        * Protected with mmap_sem in write mode as multiple tasks
-        * might race to install the same page.
-        */
-       mmap_write_lock(alloc->mm);
-       if (binder_get_installed_page(lru_page)) {
-               ret = 1;
-               goto out;
-       }
+       mmap_read_lock(alloc->mm);
 
-       if (!alloc->vma) {
+       /* vma might have been dropped or deattached */
+       if (!alloc->vma || !find_vma(alloc->mm, addr)) {
                pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
                ret = -ESRCH;
                goto out;
drivers/android/binder_alloc.c +0 -0
@@ -257,18 +250,27 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
                goto out;
        }
 
Quoted 3 lines

drivers/android/binder_alloc.c static int binder_install_single_page(struct · line 254

Quoted 1 line
drivers/android/binder_alloc.c +1 -0
+       spin_lock(&alloc->lock);

You can't hold a spinlock and then call vm_insert_page().

Thanks! This patch has only passed the build test so far. It seems like we can hold off on further testing for now.

Quoted 29 lines
Quoted 25 lines
drivers/android/binder_alloc.c +9 -1
+       if (binder_get_installed_page(lru_page)) {
+               spin_unlock(&alloc->lock);
+               ret = 1;
+               goto out;
+       }
+
        ret = vm_insert_page(alloc->vma, addr, page);
        if (ret) {
                pr_err("%d: %s failed to insert page at offset %lx with %d\n",
                       alloc->pid, __func__, addr - alloc->buffer, ret);
+               spin_unlock(&alloc->lock);
                ret = -ENOMEM;
                goto out;
        }
 
        /* Mark page installation complete and safe to use */
        binder_set_installed_page(lru_page, page);
+       spin_unlock(&alloc->lock);
 out:
-       mmap_write_unlock(alloc->mm);
+       mmap_read_unlock(alloc->mm);
        mmput_async(alloc->mm);
        if (ret && page)
                __free_page(page);
Quoted 2 lines

--
2.39.3 (Apple Git-146)

Quoted 4 lines

Sorry, but as I mentioned, I've been working on fixing this contention by supporting concurrent "faults" in binder_install_single_page(). This is the appropriate fix. I should be sending a patch soon after working out the conflicts with the shrinker's callback.

Awesome! I’m eager to see your patch, and we’re ready to help with testing. I strongly recommend dropping the write lock entirely. Using mmap_write_lock() isn’t just a binder-specific concern; it has the potential to affect the entire Android system.

In patch 3, I experimented with using per_vma_lock as well. I’m _not_ proposing it for merging since you’re already working on it, but I wanted to share the idea. (just like patch2, it has only passed build-test)

[PATCH] binder_alloc: Further move to per_vma_lock from mmap_read_lock

To further reduce the read lock duration, let's try using per_vma_lock first. If that fails, we can take the read lock, similar to how page fault handlers operate.

---
 drivers/android/binder_alloc.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/android/binder_alloc.c +0 -0
index a2281dfacbbc..b40a5dd650c8 100644
@@ -221,6 +221,8 @@ static int binder_install_single_page(struct

binder_alloc *alloc,
struct binder_lru_page *lru_page, unsigned long addr)
{
+ struct vm_area_struct *vma;

+       bool per_vma_lock = true;
        struct page *page;
        int ret = 0;

@@ -235,10 +237,15 @@ static int binder_install_single_page(struct binder_alloc *alloc,

         */
        page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
-       mmap_read_lock(alloc->mm);
+       vma = lock_vma_under_rcu(alloc->mm, addr);
+       if (!vma) {
+               per_vma_lock = false;
+               mmap_read_lock(alloc->mm);
+               vma = find_vma(alloc->mm, addr);

+ }

-       /* vma might have been dropped or deattached */
-       if (!alloc->vma || !find_vma(alloc->mm, addr)) {
+       /* vma might have been dropped, deattached or changed to new one */
+       if (!alloc->vma || !vma || vma != alloc->vma) {
                pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
                ret = -ESRCH;
                goto out;
@@ -270,7 +277,10 @@ static int binder_install_single_page(struct
binder_alloc *alloc,
        binder_set_installed_page(lru_page, page);
        spin_unlock(&alloc->lock);

out:
- mmap_read_unlock(alloc->mm);
+ if (per_vma_lock)
+ vma_end_read(vma);
+ else

+               mmap_read_unlock(alloc->mm);
        mmput_async(alloc->mm);
        if (ret && page)
                __free_page(page);
Signature · 1 line
2.39.3 (Apple Git-146)
Quoted 4 lines

Thanks,
--
Carlos Llamas

Thanks
Barry

maintainer
#9 lore raw
Quoted 4 lines · Barry Song

However, I'm not entirely convinced that this is a problem :-) Concurrent allocations like this can occur in many places, especially in PFs. Reclamation is not useless because it helps free up memory for others; it's not without value.

Yeah, for binder though there is a high chance of multiple callers trying to allocate the _same_ page concurrently. What I observed is that pages being reclaimed "in vain" are often in the same vma and this means subsequent callers will need to allocate these pages.

But even if this wasn't an issue, the correct solution would still be to support concurrent faults. So, in reality it doesn't matter and we still need to refactor the call to be non-exclusive.

Quoted 2 lines

I also don't believe binder is one of the largest users executing concurrent allocations.

Oh, I have no statistics on this.

Quoted 8 lines

Awesome! I’m eager to see your patch, and we’re ready to help with testing. I strongly recommend dropping the write lock entirely. Using mmap_write_lock() isn’t just a binder-specific concern; it has the potential to affect the entire Android system.

In patch 3, I experimented with using per_vma_lock as well. I’m _not_ proposing it for merging since you’re already working on it, but I wanted to share the idea. (just like patch2, it has only passed build-test)

Yeap, I'm using the per-vma-lock too per Suren's suggestion.

Quoted 58 lines

[PATCH] binder_alloc: Further move to per_vma_lock from mmap_read_lock

To further reduce the read lock duration, let's try using per_vma_lock first. If that fails, we can take the read lock, similar to how page fault handlers operate.

---
 drivers/android/binder_alloc.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/android/binder_alloc.c +10 -3
index a2281dfacbbc..b40a5dd650c8 100644
@@ -221,6 +221,8 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
                                      struct binder_lru_page *lru_page,
                                      unsigned long addr)
 {
+       struct vm_area_struct *vma;
+       bool per_vma_lock = true;
        struct page *page;
        int ret = 0;
@@ -235,10 +237,15 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
         */
        page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
 
-       mmap_read_lock(alloc->mm);
+       vma = lock_vma_under_rcu(alloc->mm, addr);
+       if (!vma) {
+               per_vma_lock = false;
+               mmap_read_lock(alloc->mm);
+               vma = find_vma(alloc->mm, addr);
+       }
 
-       /* vma might have been dropped or deattached */
-       if (!alloc->vma || !find_vma(alloc->mm, addr)) {
+       /* vma might have been dropped, deattached or changed to new one */
+       if (!alloc->vma || !vma || vma != alloc->vma) {
                pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
                ret = -ESRCH;
                goto out;
drivers/android/binder_alloc.c +4 -1
@@ -270,7 +277,10 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
        binder_set_installed_page(lru_page, page);
        spin_unlock(&alloc->lock);
 out:
-       mmap_read_unlock(alloc->mm);
+       if (per_vma_lock)
+               vma_end_read(vma);
+       else
+               mmap_read_unlock(alloc->mm);
        mmput_async(alloc->mm);
        if (ret && page)
Quoted 3 lines
                __free_page(page);
Signature · 1 line
2.39.3 (Apple Git-146)

This looks fairly similar to my patch. However, you would need to handle the case were vm_insert_page() returns -EBUSY (success that raced) and also sync with the shrinker callbacks in binder_alloc_free_page() which is the biggest concern.

Let's not duplicate efforts though. Can we please wait for my patch? I'll add you as Co-Developed-by, since you've posted this already?

Regards,
Carlos Llamas

author
#10 lore raw
Quoted 99 lines · Carlos Llamas
Quoted 4 lines · Barry Song

However, I'm not entirely convinced that this is a problem :-) Concurrent allocations like this can occur in many places, especially in PFs. Reclamation is not useless because it helps free up memory for others; it's not without value.

Yeah, for binder though there is a high chance of multiple callers trying to allocate the _same_ page concurrently. What I observed is that pages being reclaimed "in vain" are often in the same vma and this means subsequent callers will need to allocate these pages.

But even if this wasn't an issue, the correct solution would still be to support concurrent faults. So, in reality it doesn't matter and we still need to refactor the call to be non-exclusive.

Quoted 2 lines

I also don't believe binder is one of the largest users executing concurrent allocations.

Oh, I have no statistics on this.

Quoted 8 lines

Awesome! I’m eager to see your patch, and we’re ready to help with testing. I strongly recommend dropping the write lock entirely. Using mmap_write_lock() isn’t just a binder-specific concern; it has the potential to affect the entire Android system.

In patch 3, I experimented with using per_vma_lock as well. I’m _not_ proposing it for merging since you’re already working on it, but I wanted to share the idea. (just like patch2, it has only passed build-test)

Yeap, I'm using the per-vma-lock too per Suren's suggestion.

Quoted 59 lines

[PATCH] binder_alloc: Further move to per_vma_lock from mmap_read_lock

To further reduce the read lock duration, let's try using per_vma_lock first. If that fails, we can take the read lock, similar to how page fault handlers operate.

---
 drivers/android/binder_alloc.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/android/binder_alloc.c +10 -3
index a2281dfacbbc..b40a5dd650c8 100644
@@ -221,6 +221,8 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
                                      struct binder_lru_page *lru_page,
                                      unsigned long addr)
 {
+       struct vm_area_struct *vma;
+       bool per_vma_lock = true;
        struct page *page;
        int ret = 0;
@@ -235,10 +237,15 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
         */
        page = alloc_page(GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO);
 
-       mmap_read_lock(alloc->mm);
+       vma = lock_vma_under_rcu(alloc->mm, addr);
+       if (!vma) {
+               per_vma_lock = false;
+               mmap_read_lock(alloc->mm);
+               vma = find_vma(alloc->mm, addr);
+       }
 
-       /* vma might have been dropped or deattached */
-       if (!alloc->vma || !find_vma(alloc->mm, addr)) {
+       /* vma might have been dropped, deattached or changed to new one */
+       if (!alloc->vma || !vma || vma != alloc->vma) {
                pr_err("%d: %s failed, no vma\n", alloc->pid, __func__);
                ret = -ESRCH;
                goto out;
drivers/android/binder_alloc.c +4 -1
@@ -270,7 +277,10 @@ static int binder_install_single_page(struct
 binder_alloc *alloc,
        binder_set_installed_page(lru_page, page);
        spin_unlock(&alloc->lock);
 out:
-       mmap_read_unlock(alloc->mm);
+       if (per_vma_lock)
+               vma_end_read(vma);
+       else
+               mmap_read_unlock(alloc->mm);
        mmput_async(alloc->mm);
        if (ret && page)
                __free_page(page);
Quoted 2 lines

--
2.39.3 (Apple Git-146)

This looks fairly similar to my patch. However, you would need to handle the case were vm_insert_page() returns -EBUSY (success that raced) and also sync with the shrinker callbacks in binder_alloc_free_page() which is the biggest concern.

Quoted 2 lines

Let's not duplicate efforts though. Can we please wait for my patch? I'll add you as Co-Developed-by, since you've posted this already?

Yes, I’d be more than happy to wait for your patch, as I believe you have much much more experience with binder.

Quoted 3 lines

Regards,
Carlos Llamas

Thanks
Barry