Skip to contentSkip to conversation

rapidio: Fix error handling path

PATCH Applied Souptick Joarder 4 participants 19 messages Sep 16, 2020 – Sep 21, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rapidio: Fix error handling path

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

Fixes e8de370188d09 ("rapidio: add mport char device driver")
Cc Ira Weiny <[email protected]>
Cc John Hubbard <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 drivers/rapidio/devices/rio_mport_cdev.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
drivers/rapidio/devices/rio_mport_cdev.c +7 -6
index a303429..163b6c72 100644
@@ -871,15 +871,16 @@ static int do_dma_request(struct mport_dma_req *req,
 				rmcd_error("pin_user_pages_fast err=%ld",
 					   pinned);
 				nr_pages = 0;
-			} else
+			} else {
 				rmcd_error("pinned %ld out of %ld pages",
 					   pinned, nr_pages);
+				/*
+				 * Set nr_pages up to mean "how many pages to unpin, in
+				 * the error handler:
+				 */
+				nr_pages = pinned;
+			}
 			ret = -EFAULT;
-			/*
-			 * Set nr_pages up to mean "how many pages to unpin, in
-			 * the error handler:
-			 */
-			nr_pages = pinned;
 			goto err_pg;
 		}
 
Signature · 1 line
1.9.1
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 4 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

Quoted 1 line

This patch will fix the problem.

How about:

rio_dma_transfer() attempts to clamp the return value of pin_user_pages_fast() to be >= 0. However, the attempt fails because nr_pages is overridden a few lines later, and restored to the undesirable -ERRNO value.

The return value is ultimately stored in nr_pages, which in turn is passed to unpin_user_pages(), which expects nr_pages >= 0, else, disaster.

Fix this by fixing the nesting of the assignment to nr_pages: nr_pages should be clamped to zero if pin_user_pages_fast() returns -ERRNO, or set to the return value of pin_user_pages_fast(), otherwise.

Quoted 28 lines
Fixes e8de370188d09 ("rapidio: add mport char device driver")
Cc Ira Weiny <[email protected]>
Cc John Hubbard <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
  drivers/rapidio/devices/rio_mport_cdev.c | 13 +++++++------
  1 file changed, 7 insertions(+), 6 deletions(-)
drivers/rapidio/devices/rio_mport_cdev.c +6 -1
index a303429..163b6c72 100644
@@ -871,15 +871,16 @@ static int do_dma_request(struct mport_dma_req *req,
  				rmcd_error("pin_user_pages_fast err=%ld",
  					   pinned);
  				nr_pages = 0;
-			} else
+			} else {
  				rmcd_error("pinned %ld out of %ld pages",
  					   pinned, nr_pages);
+				/*
+				 * Set nr_pages up to mean "how many pages to unpin, in
+				 * the error handler:
+				 */
+				nr_pages = pinned;

drivers/rapidio/devices/rio_mport_cdev.c static int do_dma_request(struct mport_dma_req *req, · line 886

Quoted 10 lines
drivers/rapidio/devices/rio_mport_cdev.c +1 -5
+			}
  			ret = -EFAULT;
-			/*
-			 * Set nr_pages up to mean "how many pages to unpin, in
-			 * the error handler:
-			 */
-			nr_pages = pinned;
  			goto err_pg;
  		}
 

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#2 lore raw
In reply to Souptick Joarder
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

	   int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);

3) We could add a check in unpin_user_pages().

	if (WARN_ON(IS_ERR_VALUE(npages)))
		return;

It's not possible to pin more than "ULONG_MAX - 4095" because otherwise returning error pointers wouldn't work. So this check can't break anything and it could prevent a crash.

regards,
dan carpenter

#3 lore raw
Quoted 23 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

	   int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);
Quoted 8 lines

3) We could add a check in unpin_user_pages().

	if (WARN_ON(IS_ERR_VALUE(npages)))
		return;

It's not possible to pin more than "ULONG_MAX - 4095" because otherwise returning error pointers wouldn't work. So this check can't break anything and it could prevent a crash.

Actually pin_user_pages_fast() returns an int. It's not possible to pin more than INT_MAX.

regards,
dan carpenter

Reviewed-by on v1 1/1
#18 lore raw
In reply to Souptick Joarder
Quoted 9 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

Fixes e8de370188d09 ("rapidio: add mport char device driver")
Cc Ira Weiny <[email protected]>
Quoted 36 lines
Cc John Hubbard <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 drivers/rapidio/devices/rio_mport_cdev.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
drivers/rapidio/devices/rio_mport_cdev.c +7 -6
index a303429..163b6c72 100644
@@ -871,15 +871,16 @@ static int do_dma_request(struct mport_dma_req *req,
 				rmcd_error("pin_user_pages_fast err=%ld",
 					   pinned);
 				nr_pages = 0;
-			} else
+			} else {
 				rmcd_error("pinned %ld out of %ld pages",
 					   pinned, nr_pages);
+				/*
+				 * Set nr_pages up to mean "how many pages to unpin, in
+				 * the error handler:
+				 */
+				nr_pages = pinned;
+			}
 			ret = -EFAULT;
-			/*
-			 * Set nr_pages up to mean "how many pages to unpin, in
-			 * the error handler:
-			 */
-			nr_pages = pinned;
 			goto err_pg;
 		}
 
Signature · 1 line
1.9.1
#5 lore raw
In reply to Ira Weiny
Quoted 24 lines · Ira Weiny
Quoted 23 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

	   int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);
Quoted 7 lines
Quoted 4 lines

3) We could add a check in unpin_user_pages().

	if (WARN_ON(IS_ERR_VALUE(npages)))
		return;

Does IS_ERR_VALUE() work on an unsigned variable? The issue with adding a check in unpin_user_pages() is that npages is unsigned long.

Yeah, it does. It only works on long and unsigned long.

(Or long long types if they have the same number of bytes as a long).

regards,
dan carpenter

#4 lore raw
In reply to Dan Carpenter
Quoted 21 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

Quoted 6 lines
	   int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);

3) We could add a check in unpin_user_pages().

	if (WARN_ON(IS_ERR_VALUE(npages)))
		return;

Does IS_ERR_VALUE() work on an unsigned variable? The issue with adding a check in unpin_user_pages() is that npages is unsigned long.

Ira

Quoted 8 lines

It's not possible to pin more than "ULONG_MAX - 4095" because otherwise returning error pointers wouldn't work. So this check can't break anything and it could prevent a crash.

regards,
dan carpenter

reviewer
#6 lore raw

PATCH 1/1 mm/gup: protect unpin_user_pages() against npages==-ERRNO

In reply to Dan Carpenter

As suggested by Dan Carpenter, fortify unpin_user_pages() just a bit, against a typical caller mistake: check if the npages arg is really a -ERRNO value, which would blow up the unpinning loop: WARN and return.

If this new WARN_ON() fires, then the system *might* be leaking pages (by leaving them pinned), but probably not. More likely, gup/pup returned a hard -ERRNO error to the caller, who erroneously passed it here.

---

Hi Dan,

Is is OK to use your signed-off-by here? Since you came up with this.

thanks,
John Hubbard
NVIDIA

 mm/gup.c | 7 +++++++
 1 file changed, 7 insertions(+)
mm/gup.c +7 -0
index e5739a1974d5..41d082707016 100644
@@ -328,6 +328,13 @@ void unpin_user_pages(struct page **pages, unsigned long npages)
 {
 	unsigned long index;
 
+	/*
+	 * If this WARN_ON() fires, then the system *might* be leaking pages (by
+	 * leaving them pinned), but probably not. More likely, gup/pup returned
+	 * a hard -ERRNO error to the caller, who erroneously passed it here.
+	 */
+	if (WARN_ON(IS_ERR_VALUE(npages)))
+		return;
 	/*
 	 * TODO: this can be optimized for huge pages: if a series of pages is
 	 * physically contiguous and part of the same compound page, then a
Signature · 1 line
2.28.0
#7 lore raw

Re: [PATCH] mm/gup: protect unpin_user_pages() against npages==-ERRNO

Quoted 13 lines · John Hubbard

As suggested by Dan Carpenter, fortify unpin_user_pages() just a bit, against a typical caller mistake: check if the npages arg is really a -ERRNO value, which would blow up the unpinning loop: WARN and return.

If this new WARN_ON() fires, then the system *might* be leaking pages (by leaving them pinned), but probably not. More likely, gup/pup returned a hard -ERRNO error to the caller, who erroneously passed it here.

Quoted 6 lines

---

Hi Dan,

Is is OK to use your signed-off-by here? Since you came up with this.

Yeah. That's fine.

regards,
dan carpenter

#11 lore raw
In reply to Dan Carpenter
Quoted 14 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

Quoted 9 lines

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

	   int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);

I only found one bug but I'm going to add unpin_user_pages_dirty_lock() to the mix a retest. There were a few other false positives. In reviewing the code, I noticed that orangefs_bufmap_map() is also buggy.

I sort of feel like returning partial successes is not working. We could easily make a wrapper which either pins everything or it returns an error code.

drivers/misc/mic/scif/scif_rma.c:1399 __scif_pin_pages() warn: unpinning negative pages 'pinned_pages->nr_pages'

drivers/misc/mic/scif/scif_rma.c
1355 vmalloc_addr = true;
1356
1357 for (i = 0; i < nr_pages; i++) {
1358 if (vmalloc_addr)
1359 pinned_pages->pages[i] =
1360 vmalloc_to_page(addr + (i * PAGE_SIZE));
1361 else
1362 pinned_pages->pages[i] =
1363 virt_to_page(addr + (i * PAGE_SIZE));

  1364                  }
  1365                  pinned_pages->nr_pages = nr_pages;
  1366                  pinned_pages->map_flags = SCIF_MAP_KERNEL;
  1367          } else {
  1368                  /*
  1369                   * SCIF supports registration caching. If a registration has
  1370                   * been requested with read only permissions, then we try
  1371                   * to pin the pages with RW permissions so that a subsequent
  1372                   * transfer with RW permission can hit the cache instead of
  1373                   * invalidating it. If the upgrade fails with RW then we
  1374                   * revert back to R permission and retry
  1375                   */
  1376                  if (prot == SCIF_PROT_READ)
  1377                          try_upgrade = true;
  1378                  prot |= SCIF_PROT_WRITE;
  1379  retry:
  1380                  mm = current->mm;
  1381                  if (ulimit) {
  1382                          err = __scif_check_inc_pinned_vm(mm, nr_pages);
  1383                          if (err) {
  1384                                  pinned_pages->nr_pages = 0;
  1385                                  goto error_unmap;
  1386                          }
  1387                  }
  1388  
  1389                  pinned_pages->nr_pages = pin_user_pages_fast(
  1390                                  (u64)addr,
  1391                                  nr_pages,
  1392                                  (prot & SCIF_PROT_WRITE) ? FOLL_WRITE : 0,
  1393                                  pinned_pages->pages);
  1394                  if (nr_pages != pinned_pages->nr_pages) {
  1395                          if (try_upgrade) {
  1396                                  if (ulimit)
  1397                                          __scif_dec_pinned_vm_lock(mm, nr_pages);
  1398                                  /* Roll back any pinned pages */
  1399                                  unpin_user_pages(pinned_pages->pages,
  1400                                                   pinned_pages->nr_pages);
                                                         ^^^^^^^^^^^^^^^^^^^^^^

Negative.

  1401                                  prot &= ~SCIF_PROT_WRITE;
  1402                                  try_upgrade = false;
  1403                                  goto retry;
  1404                          }
  1405                  }
  1406                  pinned_pages->map_flags = 0;
  1407          }
  1408  
  1409          if (pinned_pages->nr_pages < nr_pages) {
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These are both signed so it negative ->nr_pages are less than nr_pages.

1410 err = -EFAULT;

  1411                  pinned_pages->nr_pages = nr_pages;
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This sets it to "everything was pinned".

1412 goto dec_pinned;
1413 }
1414

  1415          *out_prot = prot;
  1416          atomic_set(&pinned_pages->ref_count, 1);
  1417          *pages = pinned_pages;
  1418          return err;
  1419  dec_pinned:
  1420          if (ulimit)
  1421                  __scif_dec_pinned_vm_lock(mm, nr_pages);
  1422          /* Something went wrong! Rollback */
  1423  error_unmap:
  1424          pinned_pages->nr_pages = nr_pages;
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This assumes everything was pinned successfully.

  1425          scif_destroy_pinned_pages(pinned_pages);
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We absolutely don't want to pass negative ->nr_pages to this function either.

  1426          *pages = NULL;
  1427          dev_dbg(scif_info.mdev.this_device,
  1428                  "%s %d err %d len 0x%lx\n", __func__, __LINE__, err, len);
  1429          return err;
  1430  }

regards,
dan carpenter

#12 lore raw
Quoted 25 lines · Dan Carpenter
Quoted 14 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

Quoted 9 lines

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

	   int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);
Quoted 7 lines

I only found one bug but I'm going to add unpin_user_pages_dirty_lock() to the mix a retest. There were a few other false positives. In reviewing the code, I noticed that orangefs_bufmap_map() is also buggy.

I sort of feel like returning partial successes is not working. We could easily make a wrapper which either pins everything or it returns an error code.

I guess the question is are there drivers which will keep working (or limp along?) on partial pins? A quick search of a driver I thought did this does not apparently any more... So it sounds good to me from 30,000 feet! :-D

Ira

Quoted 100 lines

drivers/misc/mic/scif/scif_rma.c:1399 __scif_pin_pages() warn: unpinning negative pages 'pinned_pages->nr_pages'

drivers/misc/mic/scif/scif_rma.c
1355 vmalloc_addr = true;
1356
1357 for (i = 0; i < nr_pages; i++) {
1358 if (vmalloc_addr)
1359 pinned_pages->pages[i] =
1360 vmalloc_to_page(addr + (i * PAGE_SIZE));
1361 else
1362 pinned_pages->pages[i] =
1363 virt_to_page(addr + (i * PAGE_SIZE));

  1364                  }
  1365                  pinned_pages->nr_pages = nr_pages;
  1366                  pinned_pages->map_flags = SCIF_MAP_KERNEL;
  1367          } else {
  1368                  /*
  1369                   * SCIF supports registration caching. If a registration has
  1370                   * been requested with read only permissions, then we try
  1371                   * to pin the pages with RW permissions so that a subsequent
  1372                   * transfer with RW permission can hit the cache instead of
  1373                   * invalidating it. If the upgrade fails with RW then we
  1374                   * revert back to R permission and retry
  1375                   */
  1376                  if (prot == SCIF_PROT_READ)
  1377                          try_upgrade = true;
  1378                  prot |= SCIF_PROT_WRITE;
  1379  retry:
  1380                  mm = current->mm;
  1381                  if (ulimit) {
  1382                          err = __scif_check_inc_pinned_vm(mm, nr_pages);
  1383                          if (err) {
  1384                                  pinned_pages->nr_pages = 0;
  1385                                  goto error_unmap;
  1386                          }
  1387                  }
  1388  
  1389                  pinned_pages->nr_pages = pin_user_pages_fast(
  1390                                  (u64)addr,
  1391                                  nr_pages,
  1392                                  (prot & SCIF_PROT_WRITE) ? FOLL_WRITE : 0,
  1393                                  pinned_pages->pages);
  1394                  if (nr_pages != pinned_pages->nr_pages) {
  1395                          if (try_upgrade) {
  1396                                  if (ulimit)
  1397                                          __scif_dec_pinned_vm_lock(mm, nr_pages);
  1398                                  /* Roll back any pinned pages */
  1399                                  unpin_user_pages(pinned_pages->pages,
  1400                                                   pinned_pages->nr_pages);
                                                         ^^^^^^^^^^^^^^^^^^^^^^

Negative.

  1401                                  prot &= ~SCIF_PROT_WRITE;
  1402                                  try_upgrade = false;
  1403                                  goto retry;
  1404                          }
  1405                  }
  1406                  pinned_pages->map_flags = 0;
  1407          }
  1408  
  1409          if (pinned_pages->nr_pages < nr_pages) {
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These are both signed so it negative ->nr_pages are less than nr_pages.

1410 err = -EFAULT;

  1411                  pinned_pages->nr_pages = nr_pages;
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This sets it to "everything was pinned".

1412 goto dec_pinned;
1413 }
1414

  1415          *out_prot = prot;
  1416          atomic_set(&pinned_pages->ref_count, 1);
  1417          *pages = pinned_pages;
  1418          return err;
  1419  dec_pinned:
  1420          if (ulimit)
  1421                  __scif_dec_pinned_vm_lock(mm, nr_pages);
  1422          /* Something went wrong! Rollback */
  1423  error_unmap:
  1424          pinned_pages->nr_pages = nr_pages;
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This assumes everything was pinned successfully.

  1425          scif_destroy_pinned_pages(pinned_pages);
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We absolutely don't want to pass negative ->nr_pages to this function either.

  1426          *pages = NULL;
  1427          dev_dbg(scif_info.mdev.this_device,
  1428                  "%s %d err %d len 0x%lx\n", __func__, __LINE__, err, len);
  1429          return err;
  1430  }

regards,
dan carpenter

reviewer
#13 lore raw
Quoted 26 lines · Ira Weiny
Quoted 25 lines · Dan Carpenter
Quoted 23 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);

Quoted 7 lines
Quoted 7 lines

I only found one bug but I'm going to add unpin_user_pages_dirty_lock() to the mix a retest. There were a few other false positives. In reviewing the code, I noticed that orangefs_bufmap_map() is also buggy.

I sort of feel like returning partial successes is not working. We could easily make a wrapper which either pins everything or it returns an error code.

Yes we could. And I have the same feeling about this API. It's generated a remarkable amount of bug fixes, several of which ended up being partial or wrong in themselves. And mostly this is due to the complicated tristate return code: instead of 0 or -ERRNO, it also can return "N pages that is less than what you requested", and there are no standard helpers in the kernel to make that easier to deal with.

Quoted 4 lines

I guess the question is are there drivers which will keep working (or limp along?) on partial pins? A quick search of a driver I thought did this does not apparently any more... So it sounds good to me from 30,000 feet! :-D

It sounds good to me too--and from just a *few hundred feet* (having touched most of the call sites at some point)! haha :)

I think the wrapper should be short-term, though, just until all the callers are converted to the simpler API. Then change the core gup/pup calls to the simpler API. There are more than enough gup/pup API entry points as it is, that's for sure.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#14 lore raw
Quoted 36 lines · John Hubbard
Quoted 26 lines · Ira Weiny
Quoted 25 lines · Dan Carpenter
Quoted 23 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);

Quoted 7 lines
Quoted 7 lines

I only found one bug but I'm going to add unpin_user_pages_dirty_lock() to the mix a retest. There were a few other false positives. In reviewing the code, I noticed that orangefs_bufmap_map() is also buggy.

I sort of feel like returning partial successes is not working. We could easily make a wrapper which either pins everything or it returns an error code.

Quoted 6 lines

Yes we could. And I have the same feeling about this API. It's generated a remarkable amount of bug fixes, several of which ended up being partial or wrong in themselves. And mostly this is due to the complicated tristate return code: instead of 0 or -ERRNO, it also can return "N pages that is less than what you requested", and there are no standard helpers in the kernel to make that easier to deal with

There was some discussion on removing return value 0 from one of the gup variants [1].
I think it might be partially relevant to the current discussion.

[1] https://patchwork.kernel.org/patch/11529795/

Quoted 18 lines
Quoted 5 lines

I guess the question is are there drivers which will keep working (or limp along?) on partial pins? A quick search of a driver I thought did this does not apparently any more... So it sounds good to me from 30,000 feet! :-D

It sounds good to me too--and from just a *few hundred feet* (having touched most of the call sites at some point)! haha :)

I think the wrapper should be short-term, though, just until all the callers are converted to the simpler API. Then change the core gup/pup calls to the simpler API. There are more than enough gup/pup API entry points as it is, that's for sure.

thanks,
--
John Hubbard
NVIDIA

#16 lore raw
In reply to Dan Carpenter

Hi Dan,

Quoted 124 lines · Dan Carpenter
Quoted 14 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

Quoted 9 lines

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

         int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);

I only found one bug but I'm going to add unpin_user_pages_dirty_lock() to the mix a retest. There were a few other false positives. In reviewing the code, I noticed that orangefs_bufmap_map() is also buggy.

I sort of feel like returning partial successes is not working. We could easily make a wrapper which either pins everything or it returns an error code.

drivers/misc/mic/scif/scif_rma.c:1399 __scif_pin_pages() warn: unpinning negative pages 'pinned_pages->nr_pages'

drivers/misc/mic/scif/scif_rma.c
1355 vmalloc_addr = true;
1356
1357 for (i = 0; i < nr_pages; i++) {
1358 if (vmalloc_addr)
1359 pinned_pages->pages[i] =
1360 vmalloc_to_page(addr + (i * PAGE_SIZE));
1361 else
1362 pinned_pages->pages[i] =
1363 virt_to_page(addr + (i * PAGE_SIZE));

  1364                  }
  1365                  pinned_pages->nr_pages = nr_pages;
  1366                  pinned_pages->map_flags = SCIF_MAP_KERNEL;
  1367          } else {
  1368                  /*
  1369                   * SCIF supports registration caching. If a registration has
  1370                   * been requested with read only permissions, then we try
  1371                   * to pin the pages with RW permissions so that a subsequent
  1372                   * transfer with RW permission can hit the cache instead of
  1373                   * invalidating it. If the upgrade fails with RW then we
  1374                   * revert back to R permission and retry
  1375                   */
  1376                  if (prot == SCIF_PROT_READ)
  1377                          try_upgrade = true;
  1378                  prot |= SCIF_PROT_WRITE;
  1379  retry:
  1380                  mm = current->mm;
  1381                  if (ulimit) {
  1382                          err = __scif_check_inc_pinned_vm(mm, nr_pages);
  1383                          if (err) {
  1384                                  pinned_pages->nr_pages = 0;
  1385                                  goto error_unmap;
  1386                          }
  1387                  }
  1388
  1389                  pinned_pages->nr_pages = pin_user_pages_fast(
  1390                                  (u64)addr,
  1391                                  nr_pages,
  1392                                  (prot & SCIF_PROT_WRITE) ? FOLL_WRITE : 0,
  1393                                  pinned_pages->pages);
  1394                  if (nr_pages != pinned_pages->nr_pages) {
  1395                          if (try_upgrade) {
  1396                                  if (ulimit)
  1397                                          __scif_dec_pinned_vm_lock(mm, nr_pages);
  1398                                  /* Roll back any pinned pages */
  1399                                  unpin_user_pages(pinned_pages->pages,
  1400                                                   pinned_pages->nr_pages);
                                                         ^^^^^^^^^^^^^^^^^^^^^^

Negative.

  1401                                  prot &= ~SCIF_PROT_WRITE;
  1402                                  try_upgrade = false;
  1403                                  goto retry;
  1404                          }
  1405                  }
  1406                  pinned_pages->map_flags = 0;
  1407          }
  1408
  1409          if (pinned_pages->nr_pages < nr_pages) {
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These are both signed so it negative ->nr_pages are less than nr_pages.

1410 err = -EFAULT;

  1411                  pinned_pages->nr_pages = nr_pages;
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This sets it to "everything was pinned".

1412 goto dec_pinned;
1413 }
1414

  1415          *out_prot = prot;
  1416          atomic_set(&pinned_pages->ref_count, 1);
  1417          *pages = pinned_pages;
  1418          return err;
  1419  dec_pinned:
  1420          if (ulimit)
  1421                  __scif_dec_pinned_vm_lock(mm, nr_pages);
  1422          /* Something went wrong! Rollback */
  1423  error_unmap:
  1424          pinned_pages->nr_pages = nr_pages;
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This assumes everything was pinned successfully.

  1425          scif_destroy_pinned_pages(pinned_pages);
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We absolutely don't want to pass negative ->nr_pages to this function either.

Quoted 8 lines
  1426          *pages = NULL;
  1427          dev_dbg(scif_info.mdev.this_device,
  1428                  "%s %d err %d len 0x%lx\n", __func__, __LINE__, err, len);
  1429          return err;
  1430  }

regards,
dan carpenter

I have drafted a patch for this bug. Shall I post it ?

#17 lore raw
Quoted 133 lines · Souptick Joarder

Hi Dan,

Quoted 130 lines · Dan Carpenter
Quoted 14 lines · Dan Carpenter
Quoted 5 lines · Souptick Joarder

There is an error when pin_user_pages_fast() returns -ERRNO and inside error handling path driver end up calling unpin_user_pages() with -ERRNO which is not correct.

This patch will fix the problem.

There are a few ways we could prevent bug in the future.

1) This could have been caught with existing static analysis tools which warn about when a value is set but not used.

2) I've created a Smatch check which warngs about:

Quoted 9 lines

drivers/rapidio/devices/rio_mport_cdev.c:955 rio_dma_transfer() warn: unpinning negative pages 'nr_pages'

I'll test it out tonight and see how well it works. I don't immediately see any other bugs allthough Smatch doesn't like the code in siw_umem_release(). It uses "min_t(int" which suggests that negative pages are okay.

         int to_free = min_t(int, PAGES_PER_CHUNK, num_pages);

I only found one bug but I'm going to add unpin_user_pages_dirty_lock() to the mix a retest. There were a few other false positives. In reviewing the code, I noticed that orangefs_bufmap_map() is also buggy.

I sort of feel like returning partial successes is not working. We could easily make a wrapper which either pins everything or it returns an error code.

drivers/misc/mic/scif/scif_rma.c:1399 __scif_pin_pages() warn: unpinning negative pages 'pinned_pages->nr_pages'

drivers/misc/mic/scif/scif_rma.c
1355 vmalloc_addr = true;
1356
1357 for (i = 0; i < nr_pages; i++) {
1358 if (vmalloc_addr)
1359 pinned_pages->pages[i] =
1360 vmalloc_to_page(addr + (i * PAGE_SIZE));
1361 else
1362 pinned_pages->pages[i] =
1363 virt_to_page(addr + (i * PAGE_SIZE));

  1364                  }
  1365                  pinned_pages->nr_pages = nr_pages;
  1366                  pinned_pages->map_flags = SCIF_MAP_KERNEL;
  1367          } else {
  1368                  /*
  1369                   * SCIF supports registration caching. If a registration has
  1370                   * been requested with read only permissions, then we try
  1371                   * to pin the pages with RW permissions so that a subsequent
  1372                   * transfer with RW permission can hit the cache instead of
  1373                   * invalidating it. If the upgrade fails with RW then we
  1374                   * revert back to R permission and retry
  1375                   */
  1376                  if (prot == SCIF_PROT_READ)
  1377                          try_upgrade = true;
  1378                  prot |= SCIF_PROT_WRITE;
  1379  retry:
  1380                  mm = current->mm;
  1381                  if (ulimit) {
  1382                          err = __scif_check_inc_pinned_vm(mm, nr_pages);
  1383                          if (err) {
  1384                                  pinned_pages->nr_pages = 0;
  1385                                  goto error_unmap;
  1386                          }
  1387                  }
  1388
  1389                  pinned_pages->nr_pages = pin_user_pages_fast(
  1390                                  (u64)addr,
  1391                                  nr_pages,
  1392                                  (prot & SCIF_PROT_WRITE) ? FOLL_WRITE : 0,
  1393                                  pinned_pages->pages);
  1394                  if (nr_pages != pinned_pages->nr_pages) {
  1395                          if (try_upgrade) {
  1396                                  if (ulimit)
  1397                                          __scif_dec_pinned_vm_lock(mm, nr_pages);
  1398                                  /* Roll back any pinned pages */
  1399                                  unpin_user_pages(pinned_pages->pages,
  1400                                                   pinned_pages->nr_pages);
                                                         ^^^^^^^^^^^^^^^^^^^^^^

Negative.

  1401                                  prot &= ~SCIF_PROT_WRITE;
  1402                                  try_upgrade = false;
  1403                                  goto retry;
  1404                          }
  1405                  }
  1406                  pinned_pages->map_flags = 0;
  1407          }
  1408
  1409          if (pinned_pages->nr_pages < nr_pages) {
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These are both signed so it negative ->nr_pages are less than nr_pages.

1410 err = -EFAULT;

  1411                  pinned_pages->nr_pages = nr_pages;
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This sets it to "everything was pinned".

1412 goto dec_pinned;
1413 }
1414

  1415          *out_prot = prot;
  1416          atomic_set(&pinned_pages->ref_count, 1);
  1417          *pages = pinned_pages;
  1418          return err;
  1419  dec_pinned:
  1420          if (ulimit)
  1421                  __scif_dec_pinned_vm_lock(mm, nr_pages);
  1422          /* Something went wrong! Rollback */
  1423  error_unmap:
  1424          pinned_pages->nr_pages = nr_pages;
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This assumes everything was pinned successfully.

  1425          scif_destroy_pinned_pages(pinned_pages);
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We absolutely don't want to pass negative ->nr_pages to this function either.

  1426          *pages = NULL;
  1427          dev_dbg(scif_info.mdev.this_device,
  1428                  "%s %d err %d len 0x%lx\n", __func__, __LINE__, err, len);
  1429          return err;
  1430  }
Quoted 5 lines
Quoted 2 lines

regards,
dan carpenter

I have drafted a patch for this bug. Shall I post it ?

Absolutely, please. :)

regards,
dan carpenter

reviewer
#15 lore raw
In reply to Souptick Joarder
Quoted 1 line · Souptick Joarder

On Thu, Sep 17, 2020 at 11:17 PM John Hubbard <[email protected]> wrote:

...

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines

I sort of feel like returning partial successes is not working. We could easily make a wrapper which either pins everything or it returns an error code.

Yes we could. And I have the same feeling about this API. It's generated a remarkable amount of bug fixes, several of which ended up being partial or wrong in themselves. And mostly this is due to the complicated tristate return code: instead of 0 or -ERRNO, it also can return "N pages that is less than what you requested", and there are no standard helpers in the kernel to make that easier to deal with

Quoted 6 lines

There was some discussion on removing return value 0 from one of the gup variants [1].
I think it might be partially relevant to the current discussion.

[1] https://patchwork.kernel.org/patch/11529795/

Yes, although as I mentioned above, I'm thinking of a 0 or -ERRNO return value, and not even return nr_pages at all.

But in any case, as a practical matter, I'm not sure if it's a good idea to actually change all the callsites, or not. If we just fix the remaining buggy callers, maybe that's better than the churn associated with another API change.

On the other-other hand, there does seem to be more churn coming anyway, with talk of actually doing a [get|pin]_user_bvec(), for example. So maybe it's better to head off the coming mess.

This is something that should be discussed on linux-mm.

Quoted 14 lines
Quoted 14 lines
Quoted 5 lines

I guess the question is are there drivers which will keep working (or limp along?) on partial pins? A quick search of a driver I thought did this does not apparently any more... So it sounds good to me from 30,000 feet! :-D

It sounds good to me too--and from just a *few hundred feet* (having touched most of the call sites at some point)! haha :)

I think the wrapper should be short-term, though, just until all the callers are converted to the simpler API. Then change the core gup/pup calls to the simpler API. There are more than enough gup/pup API entry points as it is, that's for sure.

Quoted 4 lines
Quoted 4 lines

thanks,
--
John Hubbard
NVIDIA

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#8 lore raw

Re: [PATCH] mm/gup: protect unpin_user_pages() against npages==-ERRNO

In reply to Dan Carpenter
Quoted 17 lines · Dan Carpenter
Quoted 15 lines · John Hubbard

As suggested by Dan Carpenter, fortify unpin_user_pages() just a bit, against a typical caller mistake: check if the npages arg is really a -ERRNO value, which would blow up the unpinning loop: WARN and return.

If this new WARN_ON() fires, then the system *might* be leaking pages (by leaving them pinned), but probably not. More likely, gup/pup returned a hard -ERRNO error to the caller, who erroneously passed it here.

---

Quoted 6 lines
Quoted 4 lines

Hi Dan,

Is is OK to use your signed-off-by here? Since you came up with this.

Yeah. That's fine.

Do we need a similar check inside unpin_user_pages_dirty_lock(), when make_dirty set to false ?

reviewer
#9 lore raw

Re: [PATCH] mm/gup: protect unpin_user_pages() against npages==-ERRNO

Quoted 10 lines · Souptick Joarder
Quoted 9 lines · Dan Carpenter
Quoted 8 lines · John Hubbard

As suggested by Dan Carpenter, fortify unpin_user_pages() just a bit, against a typical caller mistake: check if the npages arg is really a -ERRNO value, which would blow up the unpinning loop: WARN and return.

If this new WARN_ON() fires, then the system *might* be leaking pages (by leaving them pinned), but probably not. More likely, gup/pup returned a hard -ERRNO error to the caller, who erroneously passed it here.

...

Quoted 3 lines

Do we need a similar check inside unpin_user_pages_dirty_lock(), when make_dirty set to false ?

Maybe not. This call is rarely if ever used for error handling, but rather, for finishing up a successful use of the pages.

There is a balance between protecting against buggy callers and just fixing any buggy callers. There is also a limit to how much code one can write in hopes of avoiding bugs in...code that one writes. :) Which is why static analysis, unit and regression tests, code reviews are important too.

Here, I submit that that we're about to cross the line and go too far. But if you have any examples of buggy callers for unpin_user_pages_dirty_lock(), that might shift the line.

Or maybe others feel that we haven't gone far enough yet after all?

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#10 lore raw

Re: [PATCH] mm/gup: protect unpin_user_pages() against npages==-ERRNO

Quoted 26 lines · John Hubbard
Quoted 10 lines · Souptick Joarder
Quoted 9 lines · Dan Carpenter
Quoted 8 lines · John Hubbard

As suggested by Dan Carpenter, fortify unpin_user_pages() just a bit, against a typical caller mistake: check if the npages arg is really a -ERRNO value, which would blow up the unpinning loop: WARN and return.

If this new WARN_ON() fires, then the system *might* be leaking pages (by leaving them pinned), but probably not. More likely, gup/pup returned a hard -ERRNO error to the caller, who erroneously passed it here.

...

Quoted 3 lines

Do we need a similar check inside unpin_user_pages_dirty_lock(), when make_dirty set to false ?

Maybe not. This call is rarely if ever used for error handling, but rather, for finishing up a successful use of the pages.

There is a balance between protecting against buggy callers and just fixing any buggy callers. There is also a limit to how much code one can write in hopes of avoiding bugs in...code that one writes. :) Which is why static analysis, unit and regression tests, code reviews are important too.

Quoted 3 lines

Here, I submit that that we're about to cross the line and go too far. But if you have any examples of buggy callers for unpin_user_pages_dirty_lock(), that might shift the line.

I checked for buggy uses of unpin_user_pages_dirty_lock() using Smatch and didn't find anything. (Which doesn't mean that there aren't any).

regards,
dan carpenter