Skip to contentSkip to conversation

fs/vboxsf: Replace kmap() with kmap_local_{page, folio}()

PATCH In review Sumitra Sharma 5 participants 21 messages Jun 27, 2023 – Jun 29, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs/vboxsf: Replace kmap() with kmap_local_{page, folio}()

kmap() has been deprecated in favor of the kmap_local_page() due to high cost, restricted mapping space, the overhead of a global lock for synchronization, and making the process sleep in the absence of free slots.

kmap_local_{page, folio}() is faster than kmap() and offers thread-local and CPU-local mappings, can take pagefaults in a local kmap region and preserves preemption by saving the mappings of outgoing tasks and restoring those of the incoming one during a context switch.

The difference between kmap_local_page() and kmap_local_folio() consist only in the first taking a pointer to a page and the second taking two arguments, a pointer to a folio and the byte offset within the folio which identifies the page.

The mappings are kept thread local in the functions 'vboxsf_read_folio', 'vboxsf_writepage', 'vboxsf_write_end' in file.c

---
 fs/vboxsf/file.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
fs/vboxsf/file.c +6 -6
index 572aa1c43b37..5190619bc3c5 100644
@@ -234,7 +234,7 @@ static int vboxsf_read_folio(struct file *file, struct folio *folio)
 	u8 *buf;
 	int err;
 
-	buf = kmap(page);
+	buf = kmap_local_folio(folio, off);
 
 	err = vboxsf_read(sf_handle->root, sf_handle->handle, off, &nread, buf);
 	if (err == 0) {
@@ -245,7 +245,7 @@ static int vboxsf_read_folio(struct file *file, struct folio *folio)
 		SetPageError(page);
 	}
 
-	kunmap(page);
+	kunmap_local(buf);
 	unlock_page(page);
 	return err;
 }
@@ -286,10 +286,10 @@ static int vboxsf_writepage(struct page *page, struct writeback_control *wbc)
 	if (!sf_handle)
 		return -EBADF;
 
-	buf = kmap(page);
+	buf = kmap_local_page(page);
 	err = vboxsf_write(sf_handle->root, sf_handle->handle,
 			   off, &nwrite, buf);
-	kunmap(page);
+	kunmap_local(buf);
 
 	kref_put(&sf_handle->refcount, vboxsf_handle_release);
 
@@ -320,10 +320,10 @@ static int vboxsf_write_end(struct file *file, struct address_space *mapping,
 	if (!PageUptodate(page) && copied < len)
 		zero_user(page, from + copied, len - copied);
 
-	buf = kmap(page);
+	buf = kmap_local_page(page);
 	err = vboxsf_write(sf_handle->root, sf_handle->handle,
 			   pos, &nwritten, buf + from);
-	kunmap(page);
+	kunmap_local(buf);
 
 	if (err) {
 		nwritten = 0;
Signature · 1 line
2.25.1
Reviewed-by on v1 1/1
#1 lore raw

Hi,

Quoted 15 lines · Sumitra Sharma

kmap() has been deprecated in favor of the kmap_local_page() due to high cost, restricted mapping space, the overhead of a global lock for synchronization, and making the process sleep in the absence of free slots.

kmap_local_{page, folio}() is faster than kmap() and offers thread-local and CPU-local mappings, can take pagefaults in a local kmap region and preserves preemption by saving the mappings of outgoing tasks and restoring those of the incoming one during a context switch.

The difference between kmap_local_page() and kmap_local_folio() consist only in the first taking a pointer to a page and the second taking two arguments, a pointer to a folio and the byte offset within the folio which identifies the page.

Quoted 5 lines

The mappings are kept thread local in the functions 'vboxsf_read_folio', 'vboxsf_writepage', 'vboxsf_write_end' in file.c

Thanks, patch looks good to me:

Regards,

Hans

Quoted 52 lines
---
 fs/vboxsf/file.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
fs/vboxsf/file.c +6 -6
index 572aa1c43b37..5190619bc3c5 100644
@@ -234,7 +234,7 @@ static int vboxsf_read_folio(struct file *file, struct folio *folio)
 	u8 *buf;
 	int err;
 
-	buf = kmap(page);
+	buf = kmap_local_folio(folio, off);
 
 	err = vboxsf_read(sf_handle->root, sf_handle->handle, off, &nread, buf);
 	if (err == 0) {
@@ -245,7 +245,7 @@ static int vboxsf_read_folio(struct file *file, struct folio *folio)
 		SetPageError(page);
 	}
 
-	kunmap(page);
+	kunmap_local(buf);
 	unlock_page(page);
 	return err;
 }
@@ -286,10 +286,10 @@ static int vboxsf_writepage(struct page *page, struct writeback_control *wbc)
 	if (!sf_handle)
 		return -EBADF;
 
-	buf = kmap(page);
+	buf = kmap_local_page(page);
 	err = vboxsf_write(sf_handle->root, sf_handle->handle,
 			   off, &nwrite, buf);
-	kunmap(page);
+	kunmap_local(buf);
 
 	kref_put(&sf_handle->refcount, vboxsf_handle_release);
 
@@ -320,10 +320,10 @@ static int vboxsf_write_end(struct file *file, struct address_space *mapping,
 	if (!PageUptodate(page) && copied < len)
 		zero_user(page, from + copied, len - copied);
 
-	buf = kmap(page);
+	buf = kmap_local_page(page);
 	err = vboxsf_write(sf_handle->root, sf_handle->handle,
 			   pos, &nwritten, buf + from);
-	kunmap(page);
+	kunmap_local(buf);
 
 	if (err) {
 		nwritten = 0;
#2 lore raw
Quoted 18 lines · Hans de Goede

Hi,

Quoted 15 lines · Sumitra Sharma

kmap() has been deprecated in favor of the kmap_local_page() due to high cost, restricted mapping space, the overhead of a global lock for synchronization, and making the process sleep in the absence of free slots.

kmap_local_{page, folio}() is faster than kmap() and offers thread-local and CPU-local mappings, can take pagefaults in a local kmap region and preserves preemption by saving the mappings of outgoing tasks and restoring those of the incoming one during a context switch.

The difference between kmap_local_page() and kmap_local_folio() consist only in the first taking a pointer to a page and the second taking two arguments, a pointer to a folio and the byte offset within the folio which identifies the page.

Quoted 7 lines
Quoted 5 lines

The mappings are kept thread local in the functions 'vboxsf_read_folio', 'vboxsf_writepage', 'vboxsf_write_end' in file.c

Thanks, patch looks good to me:

It doesn't look great to me, tbh. It's generally an antipattern to map the page/folio up at the top and then pass the virtual address down to the bottom. Usually we want to work in terms of physical addresses as long as possible. I see the vmmdev_hgcm_function_parameter can take physical addresses; does it work to simply use the phys_addr instead of the linear_addr? I see this commentary:

/** Deprecated Doesn't work, use PAGELIST. */
VMMDEV_HGCM_PARM_TYPE_PHYSADDR = 3,

so, um, can we use
/** Physical addresses of locked pages for a buffer. */
VMMDEV_HGCM_PARM_TYPE_PAGELIST = 10,

and convert vboxsf_read_folio() to pass the folio down to vboxsf_read() which converts it to a PAGELIST (however one does that)?

#5 lore raw
In reply to Sumitra Sharma

fs/vboxsf/file.c static int vboxsf_read_folio(struct file *file, struct folio *folio) · line 237

Quoted 7 lines · Sumitra Sharma

+++ b/fs/vboxsf/file.c

fs/vboxsf/file.c +1 -1
@@ -234,7 +234,7 @@ static int vboxsf_read_folio(struct file *file, struct folio *folio)
 	u8 *buf;
 	int err;
 
-	buf = kmap(page);
+	buf = kmap_local_folio(folio, off);

Did you test this? 'off' is the offset in the _file_. Whereas kmap_local_folio() takes the offset within the _folio_. They have different types (loff_t vs size_t) to warn you that they're different things.

#6 lore raw

Hi Matthew,

Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Sumitra Sharma

+++ b/fs/vboxsf/file.c

fs/vboxsf/file.c +0 -0
@@ -234,7 +234,7 @@ static int vboxsf_read_folio(struct file *file, struct folio *folio)
 	u8 *buf;
 	int err;
 
Quoted 7 lines

fs/vboxsf/file.c static int vboxsf_read_folio(struct file *file, struct folio *folio) · line 237

Quoted 2 lines
fs/vboxsf/file.c +1 -1
-	buf = kmap(page);
+	buf = kmap_local_folio(folio, off);

Did you test this? 'off' is the offset in the _file_. Whereas kmap_local_folio() takes the offset within the _folio_. They have different types (loff_t vs size_t) to warn you that they're different things.

Ah yes you are completely right, off is the offset in the file and buf should point to the *start* of a mapping of the page.

Regards,

Hans

#3 lore raw
In reply to Matthew Wilcox

Hi Matthew,

Quoted 37 lines · Matthew Wilcox
Quoted 18 lines · Hans de Goede

Hi,

Quoted 15 lines · Sumitra Sharma

kmap() has been deprecated in favor of the kmap_local_page() due to high cost, restricted mapping space, the overhead of a global lock for synchronization, and making the process sleep in the absence of free slots.

kmap_local_{page, folio}() is faster than kmap() and offers thread-local and CPU-local mappings, can take pagefaults in a local kmap region and preserves preemption by saving the mappings of outgoing tasks and restoring those of the incoming one during a context switch.

The difference between kmap_local_page() and kmap_local_folio() consist only in the first taking a pointer to a page and the second taking two arguments, a pointer to a folio and the byte offset within the folio which identifies the page.

Quoted 7 lines
Quoted 5 lines

The mappings are kept thread local in the functions 'vboxsf_read_folio', 'vboxsf_writepage', 'vboxsf_write_end' in file.c

Thanks, patch looks good to me:

It doesn't look great to me, tbh. It's generally an antipattern to map the page/folio up at the top and then pass the virtual address down to the bottom. Usually we want to work in terms of physical addresses as long as possible. I see the vmmdev_hgcm_function_parameter can take physical addresses; does it work to simply use the phys_addr instead of the linear_addr? I see this commentary:

/** Deprecated Doesn't work, use PAGELIST. */
VMMDEV_HGCM_PARM_TYPE_PHYSADDR = 3,

Quoted 6 lines

so, um, can we use
/** Physical addresses of locked pages for a buffer. */
VMMDEV_HGCM_PARM_TYPE_PAGELIST = 10,

and convert vboxsf_read_folio() to pass the folio down to vboxsf_read() which converts it to a PAGELIST (however one does that)?

It has been a long time since I looked at this code in detail. I don't think you can just use different types when making virtualbox hypervisor calls and then expect the hypervisor to say sure that another way to represent a memory buffer, I'll take that instead.

After I upstreamed vboxsf support VirtualBox upstream did do some further optimizations to speed up vboxsf. So there may be something there which allows passing a physical address to the hypervisor, but I don't have the time to dive into this.

When I upstreamed this the idea was that VirtualBox upstream would see the benefits of having the guest drivers upstream and would help with upstream maintenance. But unfortunately this never materialized and they are still doing their own out of tree thing even for their guest drivers.

TL;DR: for now I believe that it is best to just keep the code as is and do a straight forward folio conversion.

Regards,

Hans

#4 lore raw

Hi,

Quoted 44 lines · Hans de Goede

Hi Matthew,

Quoted 41 lines · Matthew Wilcox
Quoted 18 lines · Hans de Goede

Hi,

Quoted 15 lines · Sumitra Sharma

kmap() has been deprecated in favor of the kmap_local_page() due to high cost, restricted mapping space, the overhead of a global lock for synchronization, and making the process sleep in the absence of free slots.

kmap_local_{page, folio}() is faster than kmap() and offers thread-local and CPU-local mappings, can take pagefaults in a local kmap region and preserves preemption by saving the mappings of outgoing tasks and restoring those of the incoming one during a context switch.

The difference between kmap_local_page() and kmap_local_folio() consist only in the first taking a pointer to a page and the second taking two arguments, a pointer to a folio and the byte offset within the folio which identifies the page.

Quoted 7 lines
Quoted 5 lines

The mappings are kept thread local in the functions 'vboxsf_read_folio', 'vboxsf_writepage', 'vboxsf_write_end' in file.c

Thanks, patch looks good to me:

It doesn't look great to me, tbh. It's generally an antipattern to map the page/folio up at the top and then pass the virtual address down to the bottom. Usually we want to work in terms of physical addresses as long as possible. I see the vmmdev_hgcm_function_parameter can take physical addresses; does it work to simply use the phys_addr instead of the linear_addr? I see this commentary:

/** Deprecated Doesn't work, use PAGELIST. */
VMMDEV_HGCM_PARM_TYPE_PHYSADDR = 3,

so, um, can we use
/** Physical addresses of locked pages for a buffer. */
VMMDEV_HGCM_PARM_TYPE_PAGELIST = 10,

Quoted 8 lines
Quoted 2 lines

and convert vboxsf_read_folio() to pass the folio down to vboxsf_read() which converts it to a PAGELIST (however one does that)?

It has been a long time since I looked at this code in detail. I don't think you can just use different types when making virtualbox hypervisor calls and then expect the hypervisor to say sure that another way to represent a memory buffer, I'll take that instead.

Ok correction to this, drivers/virt/vboxguest/vboxguest_utils.c actually already translates the VMMDEV_HGCM_PARM_TYPE_LINADDR_KERNEL_IN / VMMDEV_HGCM_PARM_TYPE_LINADDR_KERNEL_OUT buffers used by vboxsf_write()/ vboxsf_read() to PAGELIST-s before passing them to the hypervisor using page_to_phys(virt_to_page()) so we map a page and then call page_to_phys(virt_to_page()) on it which indeed is quite inefficient.

That still leaves the problem that I have very little time to look into this though ...

Regards,

Hans

#9 lore raw
In reply to Sumitra Sharma

Here's a more comprehensive read_folio patch. It's not at all efficient, but then if we wanted an efficient vboxsf, we'd implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag. I can consult with anyone who wants to do all this work.

I haven't even compiled this, just trying to show the direction this should take.

fs/vboxsf/file.c +20 -15
index 2307f8037efc..f1af9a7bd3d8 100644
@@ -227,26 +227,31 @@ const struct inode_operations vboxsf_reg_iops = {
 
 static int vboxsf_read_folio(struct file *file, struct folio *folio)
 {
-	struct page *page = &folio->page;
 	struct vboxsf_handle *sf_handle = file->private_data;
-	loff_t off = page_offset(page);
-	u32 nread = PAGE_SIZE;
-	u8 *buf;
+	loff_t pos = folio_pos(folio);
+	size_t offset = 0;
 	int err;
 
-	buf = kmap(page);
+	do {
+		u8 *buf = kmap_local_folio(folio, offset);
+		u32 nread = PAGE_SIZE;
 
-	err = vboxsf_read(sf_handle->root, sf_handle->handle, off, &nread, buf);
-	if (err == 0) {
-		memset(&buf[nread], 0, PAGE_SIZE - nread);
-		flush_dcache_page(page);
-		SetPageUptodate(page);
-	} else {
-		SetPageError(page);
-	}
+		err = vboxsf_read(sf_handle->root, sf_handle->handle, pos,
+				&nread, buf);
+		if (nread < PAGE_SIZE)
+			memset(&buf[nread], 0, PAGE_SIZE - nread);
+		kunmap_local(buf);
+		if (err)
+			break;
+		offset += PAGE_SIZE;
+		pos += PAGE_SIZE;
+	} while (offset < folio_size(folio);
 
-	kunmap(page);
-	unlock_page(page);
+	if (!err) {
+		flush_dcache_folio(folio);
+		folio_mark_uptodate(folio);
+	}
+	folio_unlock(folio);
 	return err;
 }
 
#10 lore raw
Quoted 6 lines · On mercoledì 28 giugno 2023 19:15:04 CEST Matthew Wilcox wrote:

Here's a more comprehensive read_folio patch. It's not at all efficient, but then if we wanted an efficient vboxsf, we'd implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag.
I can consult with anyone who wants to do all
this work.

Interesting...

Quoted 12 lines

I haven't even compiled this, just trying to show the direction this should take.

fs/vboxsf/file.c +0 -1
index 2307f8037efc..f1af9a7bd3d8 100644
@@ -227,26 +227,31 @@ const struct inode_operations vboxsf_reg_iops = {
 
 static int vboxsf_read_folio(struct file *file, struct folio *folio)
 {
-	struct page *page = &folio->page;

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 235

Quoted 10 lines
fs/vboxsf/file.c +3 -4
 	struct vboxsf_handle *sf_handle = file->private_data;
-	loff_t off = page_offset(page);
-	u32 nread = PAGE_SIZE;
-	u8 *buf;
+	loff_t pos = folio_pos(folio);
+	size_t offset = 0;
 	int err;
 
-	buf = kmap(page);
+	do {

Please let me understand why you are calling vboxsf_read() in a loop, a PAGE_SIZE at a time.

I have had only few minutes (whereas I'd need more time) to look at this code.

If I understand the current code it reads a single page at offset zero of a folio and then memset() with zeros from &buf[nread] up to the end of the page. Then it seems that this function currently assume that the folio doesn't need to be read until "offset < folio_size(folio)" becomes false.

Does it imply that the folio is always one page sized? Doesn't it? I'm surely missing some basics...

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = {

Quoted 4 lines
fs/vboxsf/file.c +2 -1
+		u8 *buf = kmap_local_folio(folio, offset);
+		u32 nread = PAGE_SIZE;
 
-	err = vboxsf_read(sf_handle->root, sf_handle->handle, off, &nread, 

buf);

Quoted 15 lines
fs/vboxsf/file.c +8 -7
-	if (err == 0) {
-		memset(&buf[nread], 0, PAGE_SIZE - nread);
-		flush_dcache_page(page);
-		SetPageUptodate(page);
-	} else {
-		SetPageError(page);
-	}
+		err = vboxsf_read(sf_handle->root, sf_handle->handle, pos,
+				&nread, buf);
+		if (nread < PAGE_SIZE)
+			memset(&buf[nread], 0, PAGE_SIZE - nread);
+		kunmap_local(buf);
+		if (err)
+			break;
+		offset += PAGE_SIZE;

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 254

Quoted 10 lines
fs/vboxsf/file.c +7 -2
+		pos += PAGE_SIZE;
+	} while (offset < folio_size(folio);
 
-	kunmap(page);
-	unlock_page(page);
+	if (!err) {
+		flush_dcache_folio(folio);
+		folio_mark_uptodate(folio);
+	}
+	folio_unlock(folio);

Shouldn't we call folio_lock() to lock the folio to be able to unlock with folio_unlock()?

If so, I can't find any neither a folio_lock() or a page_lock() in this filesystem.

Again sorry for not understanding, can you please explain it?

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 256

Quoted 2 lines
fs/vboxsf/file.c +0 -0
 	return err;
 }

Thanks,

Fabio

#11 lore raw

On giovedì 29 giugno 2023 00:23:54 CEST Fabio M. De Francesco wrote:

[...]

Quoted 3 lines

Shouldn't we call folio_lock() to lock the folio to be able to unlock with folio_unlock()?

Sorry, I wanted to write "If so, I can't find either a folio_lock() or a page_lock() in this filesystem.".

Fabio

Quoted 8 lines

If so, I can't find any neither a folio_lock() or a page_lock() in this filesystem.

Again sorry for not understanding, can you please explain it?

Quoted 3 lines

return err;

}

#12 lore raw
Quoted 10 lines · Fabio M. De Francesco

On giovedì 29 giugno 2023 00:23:54 CEST Fabio M. De Francesco wrote:

[...]

Quoted 3 lines

Shouldn't we call folio_lock() to lock the folio to be able to unlock with folio_unlock()?

Sorry, I wanted to write "If so, I can't find either a folio_lock() or a page_lock() in this filesystem.".

Check the documentation for read_folio:

... "The filesystem should unlock the folio once the read has completed, whether it was successful or not." ...

Ira

#15 lore raw
In reply to Matthew Wilcox
Quoted 8 lines · Matthew Wilcox

Here's a more comprehensive read_folio patch. It's not at all efficient, but then if we wanted an efficient vboxsf, we'd implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag. I can consult with anyone who wants to do all this work.

I haven't even compiled this, just trying to show the direction this should take.

I'm a bit confused. Is this progressing toward having the folio passed down to vboxsf_read() or just an expanded vboxsf_read_folio which can handle larger folios as well as fix the kmap?

Ira

Quoted 52 lines
fs/vboxsf/file.c +20 -15
index 2307f8037efc..f1af9a7bd3d8 100644
@@ -227,26 +227,31 @@ const struct inode_operations vboxsf_reg_iops = {
 
 static int vboxsf_read_folio(struct file *file, struct folio *folio)
 {
-	struct page *page = &folio->page;
 	struct vboxsf_handle *sf_handle = file->private_data;
-	loff_t off = page_offset(page);
-	u32 nread = PAGE_SIZE;
-	u8 *buf;
+	loff_t pos = folio_pos(folio);
+	size_t offset = 0;
 	int err;
 
-	buf = kmap(page);
+	do {
+		u8 *buf = kmap_local_folio(folio, offset);
+		u32 nread = PAGE_SIZE;
 
-	err = vboxsf_read(sf_handle->root, sf_handle->handle, off, &nread, buf);
-	if (err == 0) {
-		memset(&buf[nread], 0, PAGE_SIZE - nread);
-		flush_dcache_page(page);
-		SetPageUptodate(page);
-	} else {
-		SetPageError(page);
-	}
+		err = vboxsf_read(sf_handle->root, sf_handle->handle, pos,
+				&nread, buf);
+		if (nread < PAGE_SIZE)
+			memset(&buf[nread], 0, PAGE_SIZE - nread);
+		kunmap_local(buf);
+		if (err)
+			break;
+		offset += PAGE_SIZE;
+		pos += PAGE_SIZE;
+	} while (offset < folio_size(folio);
 
-	kunmap(page);
-	unlock_page(page);
+	if (!err) {
+		flush_dcache_folio(folio);
+		folio_mark_uptodate(folio);
+	}
+	folio_unlock(folio);
 	return err;
 }
 
#16 lore raw
Quoted 7 lines · Ira Weiny
Quoted 6 lines · Matthew Wilcox

Here's a more comprehensive read_folio patch. It's not at all efficient, but then if we wanted an efficient vboxsf, we'd implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag. I can consult with anyone who wants to do all this work.

Quoted 6 lines
Quoted 2 lines

I haven't even compiled this, just trying to show the direction this should take.

I'm a bit confused. Is this progressing toward having the folio passed down to vboxsf_read() or just an expanded vboxsf_read_folio which can handle larger folios as well as fix the kmap?

This just handles large folios & switches to kmap_local_folio(). Doing this properly is a _lot_ of work, and needs someone who can commit to actually testing it.

#13 lore raw
In reply to Fabio M. De Francesco
Quoted 5 lines · Fabio M. De Francesco

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 235

Quoted 2 lines
fs/vboxsf/file.c +1 -1
-	buf = kmap(page);
+	do {

Please let me understand why you are calling vboxsf_read() in a loop, a PAGE_SIZE at a time.

Because kmap_local_folio() can only (guarantee to) map one page at a time. Also vboxsf_read() is only tested with a single page at a time.

Quoted 7 lines

If I understand the current code it reads a single page at offset zero of a folio and then memset() with zeros from &buf[nread] up to the end of the page. Then it seems that this function currently assume that the folio doesn't need to be read until "offset < folio_size(folio)" becomes false.

Does it imply that the folio is always one page sized? Doesn't it? I'm surely missing some basics...

vboxsf does not yet claim to support large folios, so every folio that it sees will be only a single page in size. Hopefully at some point that will change. Again, somebody would need to test that. In the meantime, if someone is going to the trouble of switching over to using the folio API, let's actually include support for large folios.

Quoted 11 lines

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 254

Quoted 7 lines
fs/vboxsf/file.c +5 -2
-	kunmap(page);
-	unlock_page(page);
+	if (!err) {
+		flush_dcache_folio(folio);
+		folio_mark_uptodate(folio);
+	}
+	folio_unlock(folio);

Shouldn't we call folio_lock() to lock the folio to be able to unlock with folio_unlock()?

Quoted 4 lines

If so, I can't find any neither a folio_lock() or a page_lock() in this filesystem.

Again sorry for not understanding, can you please explain it?

Ira gave the minimal explanation, but a slightly fuller explanation is that the folio is locked while it is being fetched from backing store. That prevents both a second thread from reading from it while another thread is bringing it uptodate, and two threads trying to bring it uptodate at the same time.

Most filesystems have an asynchronous read_folio, so you don't see the folio_unlock() in the read_folio() function; instead it's in the I/O completion path. vboxsf is synchronous.

#7 lore raw
In reply to Matthew Wilcox
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Sumitra Sharma

+++ b/fs/vboxsf/file.c

fs/vboxsf/file.c +0 -0
@@ -234,7 +234,7 @@ static int vboxsf_read_folio(struct file *file, struct folio *folio)
 	u8 *buf;
 	int err;
 
Quoted 8 lines

fs/vboxsf/file.c static int vboxsf_read_folio(struct file *file, struct folio *folio) · line 237

Quoted 2 lines
fs/vboxsf/file.c +1 -1
-	buf = kmap(page);
+	buf = kmap_local_folio(folio, off);

Did you test this? 'off' is the offset in the _file_. Whereas kmap_local_folio() takes the offset within the _folio_. They have different types (loff_t vs size_t) to warn you that they're different things.

Hi Matthew,

When creating this patch, I read and searched about the loff_t vs size_t. By mistake, I implemented it in the wrong way.

Also, I did not test it and just compiled it. I apologise for doing so.

And for the other points you have put as feedback. I will take some time to understand it. And would like to work on the changes you suggest.

Thanks & regards
Sumitra

#8 lore raw

Hi Sumitra,

Quoted 18 lines · Sumitra Sharma
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Sumitra Sharma

+++ b/fs/vboxsf/file.c

fs/vboxsf/file.c +0 -0
@@ -234,7 +234,7 @@ static int vboxsf_read_folio(struct file *file, struct folio *folio)
 	u8 *buf;
 	int err;
 
Quoted 8 lines

fs/vboxsf/file.c static int vboxsf_read_folio(struct file *file, struct folio *folio) · line 237

Quoted 2 lines
fs/vboxsf/file.c +1 -1
-	buf = kmap(page);
+	buf = kmap_local_folio(folio, off);

Did you test this? 'off' is the offset in the _file_. Whereas kmap_local_folio() takes the offset within the _folio_. They have different types (loff_t vs size_t) to warn you that they're different things.

Hi Matthew,

Quoted 7 lines

When creating this patch, I read and searched about the loff_t vs size_t. By mistake, I implemented it in the wrong way.

Also, I did not test it and just compiled it. I apologise for doing so.

And for the other points you have put as feedback. I will take some time to understand it. And would like to work on the changes you suggest.

If you work further on this please make sure that you actually test your changes. Submitting untested fs changes is a really bad idea. People don't like it when their data gets corrupted.

Note vboxsf can be tested easily by setting up a VirtualBox guest and then using the shared folder features. Note do *not* use the VirtualBox provided guest utils they contain their own out of tree vboxsf implementation and will use that.

To avoid this you could e.g. use Fedora inside the guest and install the Fedora packaged vbox guest utils which does not replace the mainline vboxsf.

Regards,

Hans

#17 lore raw
In reply to Matthew Wilcox
Quoted 50 lines · Matthew Wilcox

Here's a more comprehensive read_folio patch. It's not at all efficient, but then if we wanted an efficient vboxsf, we'd implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag. I can consult with anyone who wants to do all this work.

I haven't even compiled this, just trying to show the direction this should take.

fs/vboxsf/file.c +15 -13
index 2307f8037efc..f1af9a7bd3d8 100644
@@ -227,26 +227,31 @@ const struct inode_operations vboxsf_reg_iops = {
 
 static int vboxsf_read_folio(struct file *file, struct folio *folio)
 {
-	struct page *page = &folio->page;
 	struct vboxsf_handle *sf_handle = file->private_data;
-	loff_t off = page_offset(page);
-	u32 nread = PAGE_SIZE;
-	u8 *buf;
+	loff_t pos = folio_pos(folio);
+	size_t offset = 0;
 	int err;
 
-	buf = kmap(page);
+	do {
+		u8 *buf = kmap_local_folio(folio, offset);
+		u32 nread = PAGE_SIZE;
 
-	err = vboxsf_read(sf_handle->root, sf_handle->handle, off, &nread, buf);
-	if (err == 0) {
-		memset(&buf[nread], 0, PAGE_SIZE - nread);
-		flush_dcache_page(page);
-		SetPageUptodate(page);
-	} else {
-		SetPageError(page);
-	}
+		err = vboxsf_read(sf_handle->root, sf_handle->handle, pos,
+				&nread, buf);
+		if (nread < PAGE_SIZE)
+			memset(&buf[nread], 0, PAGE_SIZE - nread);
+		kunmap_local(buf);
+		if (err)
+			break;
+		offset += PAGE_SIZE;
+		pos += PAGE_SIZE;
+	} while (offset < folio_size(folio);
 

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 257

Quoted 10 lines
fs/vboxsf/file.c +5 -2
-	kunmap(page);
-	unlock_page(page);
+	if (!err) {
+		flush_dcache_folio(folio);
+		folio_mark_uptodate(folio);
+	}
+	folio_unlock(folio);
 	return err;
 }
 

Hi

So, after reading the comments, I understood that the problem presented by Hans and Matthew is as follows:

1) In the current code, the buffers used by vboxsf_write()/vboxsf_read() are translated to PAGELIST-s before passing to the hypervisor,
but inefficiently— it first maps a page in vboxsf_read_folio() and then calls page_to_phys(virt_to_page()) in the function hgcm_call_init_linaddr().

The inefficiency in the current implementation arises due to the unnecessary mapping of a page in vboxsf_read_folio() because the mapping output, i.e. the linear address, is used deep down in file 'drivers/virt/vboxguest/vboxguest_utils.c'. Hence, the mapping must be done in this file; to do so, the folio must be passed until this point. It can be done by adding a new member, 'struct folio *folio', in the 'struct vmmdev_hgcm_function_parameter64'.

The unused member 'phys_addr' in this struct can also be removed.

2) Expanding the vboxsf_read_folio so that it can handle larger folios. Matthew already has suggested the changes, I have to read more on this.

Parallelly I will be setting up the testing environment to test the changes.

Please let me know if I am wrong anywhere.

Thanks & regards
Sumitra

#18 lore raw

Hi,

Quoted 72 lines · Sumitra Sharma
Quoted 50 lines · Matthew Wilcox

Here's a more comprehensive read_folio patch. It's not at all efficient, but then if we wanted an efficient vboxsf, we'd implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag. I can consult with anyone who wants to do all this work.

I haven't even compiled this, just trying to show the direction this should take.

fs/vboxsf/file.c +15 -13
index 2307f8037efc..f1af9a7bd3d8 100644
@@ -227,26 +227,31 @@ const struct inode_operations vboxsf_reg_iops = {
 
 static int vboxsf_read_folio(struct file *file, struct folio *folio)
 {
-	struct page *page = &folio->page;
 	struct vboxsf_handle *sf_handle = file->private_data;
-	loff_t off = page_offset(page);
-	u32 nread = PAGE_SIZE;
-	u8 *buf;
+	loff_t pos = folio_pos(folio);
+	size_t offset = 0;
 	int err;
 
-	buf = kmap(page);
+	do {
+		u8 *buf = kmap_local_folio(folio, offset);
+		u32 nread = PAGE_SIZE;
 
-	err = vboxsf_read(sf_handle->root, sf_handle->handle, off, &nread, buf);
-	if (err == 0) {
-		memset(&buf[nread], 0, PAGE_SIZE - nread);
-		flush_dcache_page(page);
-		SetPageUptodate(page);
-	} else {
-		SetPageError(page);
-	}
+		err = vboxsf_read(sf_handle->root, sf_handle->handle, pos,
+				&nread, buf);
+		if (nread < PAGE_SIZE)
+			memset(&buf[nread], 0, PAGE_SIZE - nread);
+		kunmap_local(buf);
+		if (err)
+			break;
+		offset += PAGE_SIZE;
+		pos += PAGE_SIZE;
+	} while (offset < folio_size(folio);
 

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 257

Quoted 10 lines
fs/vboxsf/file.c +5 -2
-	kunmap(page);
-	unlock_page(page);
+	if (!err) {
+		flush_dcache_folio(folio);
+		folio_mark_uptodate(folio);
+	}
+	folio_unlock(folio);
 	return err;
 }
 

Hi

So, after reading the comments, I understood that the problem presented by Hans and Matthew is as follows:

1) In the current code, the buffers used by vboxsf_write()/vboxsf_read() are translated to PAGELIST-s before passing to the hypervisor,
but inefficiently— it first maps a page in vboxsf_read_folio() and then calls page_to_phys(virt_to_page()) in the function hgcm_call_init_linaddr().

Quoted 6 lines

The inefficiency in the current implementation arises due to the unnecessary mapping of a page in vboxsf_read_folio() because the mapping output, i.e. the linear address, is used deep down in file 'drivers/virt/vboxguest/vboxguest_utils.c'. Hence, the mapping must be done in this file; to do so, the folio must be passed until this point. It can be done by adding a new member, 'struct folio *folio', in the 'struct vmmdev_hgcm_function_parameter64'.

struct vmmdev_hgcm_function_parameter64 is defined in

include/uapi/linux/vbox_vmmdev_types.h

This is part of the userspace API of vboxguest which allows userspace to make (some) VirtualBox hypervisor request through a chardev.

You can not just go and change this struct.

Note there already is a VMMDEV_HGCM_PARM_TYPE_PAGELIST type. So you could do the conversion from folio to pagelist (see the vboxguest code for how to do this) in the vboxsf_read() code (making it take a folio pointer as arg) and then directly pass the pagelist to the vbg_hgcm_call().

Quoted 1 line

The unused member 'phys_addr' in this struct can also be removed.

Again no you can NOT just go and make changes to an uapi header.

Regards,

Hans

#19 lore raw
In reply to Sumitra Sharma
Quoted 7 lines · Sumitra Sharma
Quoted 5 lines · Matthew Wilcox

Here's a more comprehensive read_folio patch. It's not at all efficient, but then if we wanted an efficient vboxsf, we'd implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag. I can consult with anyone who wants to do all this work.

Quoted 7 lines

So, after reading the comments, I understood that the problem presented by Hans and Matthew is as follows:

1) In the current code, the buffers used by vboxsf_write()/vboxsf_read() are translated to PAGELIST-s before passing to the hypervisor,
but inefficiently— it first maps a page in vboxsf_read_folio() and then calls page_to_phys(virt_to_page()) in the function hgcm_call_init_linaddr().

It does ... and I'm not even sure that virt_to_page() works for kmapped pages. Has it been tested with a 32-bit guest with, say, 4-8GB of memory?

Quoted 6 lines

The inefficiency in the current implementation arises due to the unnecessary mapping of a page in vboxsf_read_folio() because the mapping output, i.e. the linear address, is used deep down in file 'drivers/virt/vboxguest/vboxguest_utils.c'. Hence, the mapping must be done in this file; to do so, the folio must be passed until this point. It can be done by adding a new member, 'struct folio *folio', in the 'struct vmmdev_hgcm_function_parameter64'.

That's not the way to do it (as Hans already said).

The other problem is that vboxsf_read() is synchronous. It makes the call to the host, then waits for the outcome. What we really need is a vboxsf_readahead() that looks something like this:

static void vboxsf_readahead(struct readahead_control *ractl)

{
	unsigned int nr = readahead_count(ractl);
	req = vbg_req_alloc(... something involving nr ...);
	... fill in the page array ...
	... submit the request ...
}

You also need to set up a kthread that will sit on the hgcm_wq and handle the completions that come in (where you'd call folio_mark_uptodate() if the call is successful, folio_unlock() to indicate the I/O has completed, etc, etc).

Then go back to read_folio() (which can be synchronous), and maybe factor out the parts of vboxsf_readahead() that can be reused for filling in the vbg_req.

Hans might well have better ideas about this could be structured; I'm new to the vbox code.

#20 lore raw

Hi,

Quoted 45 lines · Matthew Wilcox
Quoted 7 lines · Sumitra Sharma
Quoted 5 lines · Matthew Wilcox

Here's a more comprehensive read_folio patch. It's not at all efficient, but then if we wanted an efficient vboxsf, we'd implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag. I can consult with anyone who wants to do all this work.

Quoted 7 lines

So, after reading the comments, I understood that the problem presented by Hans and Matthew is as follows:

1) In the current code, the buffers used by vboxsf_write()/vboxsf_read() are translated to PAGELIST-s before passing to the hypervisor,
but inefficiently— it first maps a page in vboxsf_read_folio() and then calls page_to_phys(virt_to_page()) in the function hgcm_call_init_linaddr().

It does ... and I'm not even sure that virt_to_page() works for kmapped pages. Has it been tested with a 32-bit guest with, say, 4-8GB of memory?

Quoted 6 lines

The inefficiency in the current implementation arises due to the unnecessary mapping of a page in vboxsf_read_folio() because the mapping output, i.e. the linear address, is used deep down in file 'drivers/virt/vboxguest/vboxguest_utils.c'. Hence, the mapping must be done in this file; to do so, the folio must be passed until this point. It can be done by adding a new member, 'struct folio *folio', in the 'struct vmmdev_hgcm_function_parameter64'.

That's not the way to do it (as Hans already said).

The other problem is that vboxsf_read() is synchronous. It makes the call to the host, then waits for the outcome. What we really need is a vboxsf_readahead() that looks something like this:

static void vboxsf_readahead(struct readahead_control *ractl)

{
	unsigned int nr = readahead_count(ractl);
	req = vbg_req_alloc(... something involving nr ...);
	... fill in the page array ...
	... submit the request ...
}

You also need to set up a kthread that will sit on the hgcm_wq and handle the completions that come in (where you'd call folio_mark_uptodate() if the call is successful, folio_unlock() to indicate the I/O has completed, etc, etc).

Quoted 6 lines

Then go back to read_folio() (which can be synchronous), and maybe factor out the parts of vboxsf_readahead() that can be reused for filling in the vbg_req.

Hans might well have better ideas about this could be structured; I'm new to the vbox code.

I think that moving to directly passing vbox PAGELIST structs on read is something which should not be too much work.

Moving to an async model is a lot more involved and has a much larger chance of causing problems. So we need not only someone to step up to not only make the change to async, but that person also need to commit to help with maintaining vboxsf going forward, esp. wrt debug + fix any problems stemming from the move to async which are likely to only pop up much later as more and more distros move to the new code.

Regards,

Hans

#14 lore raw
In reply to Matthew Wilcox
Quoted 9 lines · On giovedì 29 giugno 2023 05:16:04 CEST Matthew Wilcox wrote:
Quoted 5 lines · Fabio M. De Francesco

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 235

Quoted 2 lines
fs/vboxsf/file.c +1 -1
-	buf = kmap(page);
+	do {

Please let me understand why you are calling vboxsf_read() in a loop, a PAGE_SIZE at a time.

Because kmap_local_folio() can only (guarantee to) map one page at a time.

Yes, one page at a time. This part served to introduce the _main_ question that is the one you answered below (i.e., since the current code maps a page a a time with no loops, do we need to manage folios spanning more than a single page?)

Quoted 3 lines

Also vboxsf_read() is only tested with a single page at a time.

Quoted 1 line

If I understand the current code it reads a single page at offset zero of

a

Quoted 8 lines
Quoted 7 lines

folio and then memset() with zeros from &buf[nread] up to the end of the page. Then it seems that this function currently assume that the folio doesn't need to be read until "offset < folio_size(folio)" becomes false.

Does it imply that the folio is always one page sized? Doesn't it? I'm surely
missing some basics...

Quoted 6 lines

vboxsf does not yet claim to support large folios, so every folio that it sees will be only a single page in size.
Hopefully at some point
that will change. Again, somebody would need to test that. In the meantime, if someone is going to the trouble of switching over to using the folio API, let's actually include support for large folios.

"[...] at some point that will change." wrt larger folios spanning multiple pages is the answer that justifies the loop. I couldn't know about this plan. Thanks for explaining that there is a plan towards that goal.

I think that Sumitra can address the task to re-use your patch to vboxsf_read_folio() and then properly test it with VirtualBox.

Instead the much larger effort to implement vboxsf_readahead() and actually do an async call with deferred setting of the uptodate flag will surely require a considerable amount of time and whoever wanted to address it would need your guidance.

You said that you are ready to provide consult, but I'm not sure whether Sumitra would be allowed to spend a large part of her time to do an out of scope task (wrt her internship).

If yes, I have nothing against. If not, I'm pretty sure that someone else can set aside enough time to address this large task ;-)

Quoted 12 lines
Quoted 12 lines

fs/vboxsf/file.c const struct inode_operations vboxsf_reg_iops = { · line 254

Quoted 8 lines
fs/vboxsf/file.c +5 -2
-	kunmap(page);
-	unlock_page(page);
+	if (!err) {
+		flush_dcache_folio(folio);
+		folio_mark_uptodate(folio);
+	}
+	folio_unlock(folio);

Shouldn't we call folio_lock() to lock the folio to be able to unlock with folio_unlock()?

Quoted 7 lines
Quoted 4 lines

If so, I can't find any neither a folio_lock() or a page_lock() in this filesystem.

Again sorry for not understanding, can you please explain it?

Ira gave the minimal explanation, but a slightly fuller explanation is that the folio is locked while it is being fetched from backing store.

This explains why I could not easily find the call to lock it.

Quoted 7 lines

That prevents both a second thread from reading from it while another thread is bringing it uptodate, and two threads trying to bring it uptodate at the same time.

Most filesystems have an asynchronous read_folio, so you don't see the folio_unlock() in the read_folio() function; instead it's in the I/O completion path. vboxsf is synchronous.

And this explains different implementation between synchronous and asynchronous reads

Again thanks,

Fabio